Fifth Edition
yield and raise syntax; the __pycache__ bytecode model; 3.3 namespace
packages; PyDoc’s all-browser mode; Unicode literal and storage
changes; and the new Windows launcher shipped with 3.3.timeit, PyPy, os.popen, generators, recursion, weak
references, __mro__, __iter__, super, __slots__, metaclasses, descriptors,
random, Sphinx, and more has been
added, along with a general increase in 2.X compatibility in both
examples and narrative.If you are learning Python for the first time and don’t need to use any existing 2.X code, I encourage you to begin with Python 3.X. It cleans up some longstanding warts in the language and trims some dated cruft, while retaining all the original core ideas and adding some nice new tools. For example, 3.X’s seamless Unicode model and broader use of generators and functional techniques are seen by many users as assets. Many popular Python libraries and tools are already available for Python 3.X, or will be by the time you read these words, especially given the continual improvements in the 3.X line. All new language evolution occurs in 3.X only, which adds features and keeps Python relevant, but also makes language definition a constantly moving target — a tradeoff inherent on the leading edge.
If you’ll be using a system based on Python 2.X, the 3.X line may not be an option for you today. However, you’ll find that this book addresses your concerns, too, and will help if you migrate to 3.X in the future. You’ll also find that you’re in large company. Every group I taught in 2012 was using 2.X only, and I still regularly see useful Python software in 2.X-only form. Moreover, unlike 3.X, 2.X is no longer being changed — which is either an asset or liability, depending on whom you ask. There’s nothing wrong with using and writing 2.X code, but you may wish to keep tabs on 3.X and its ongoing evolution as you do. Python’s future remains to be written, and is largely up to its users, including you.
Probably the best news here is that Python’s fundamentals are the same in both its lines — 2.X and 3.X differ in ways that many users will find minor, and this book is designed to help you learn both. In fact, as long as you understand their differences, it’s often straightforward to write version-neutral code that runs on both Pythons, as we regularly will in this book. See Appendix C for pointers on 2.X/3.X migration and tips on writing code for both Python lines and audiences.
You have an initial advantage and can move quickly through some earlier chapters; but you shouldn’t skip the core ideas, and may need to work at letting go of some baggage. In general terms, exposure to any programming or scripting before this book might be helpful because of the analogies it may provide. On the other hand, I’ve also found that prior programming experience can be a handicap due to expectations rooted in other languages (it’s far too easy to spot the Java or C++ programmers in classes by the first Python code they write!). Using Python well requires adopting its mindset. By focusing on key core concepts, this book is designed to help you learn to code Python in Python.
You can learn Python here too, as well as programming itself; but you may need to work a bit harder, and may wish to supplement this text with gentler introductions. If you don’t consider yourself a programmer already, you will probably find this book useful too, but you’ll want to be sure to proceed slowly and work through the examples and exercises along the way. Also keep in mind that this book will spend more time teaching Python itself than programming basics. If you find yourself lost here, I encourage you to explore an introduction to programming in general before tackling this book. Python’s website has links to many helpful resources for beginners.
We begin with a general overview of Python that answers commonly asked initial questions — why people use the language, what it’s useful for, and so on. The first chapter introduces the major ideas underlying the technology to give you some background context. The rest of this part moves on to explore the ways that both Python and programmers run programs. The main goal here is to give you just enough information to be able to follow along with later examples and exercises.
Next, we begin our tour of the Python language, studying Python’s major built-in object types and what you can do with them in depth: numbers, lists, dictionaries, and so on. You can get a lot done with these tools alone, and they are at the heart of every Python script. This is the most substantial part of the book because we lay groundwork here for later chapters. We’ll also explore dynamic typing and its references — keys to using Python well — in this part.
The next part moves on to introduce Python’s statements — the code you type to create and process objects in Python. It also presents Python’s general syntax model. Although this part focuses on syntax, it also introduces some related tools (such as the PyDoc system), takes a first look at iteration concepts, and explores coding alternatives.
This part begins our look at Python’s higher-level program structure tools. Functions turn out to be a simple way to package code for reuse and avoid code redundancy. In this part, we will explore Python’s scoping rules, argument-passing techniques, the sometimes-notorious lambda, and more. We’ll also revisit iterators from a functional programming perspective, introduce user-defined generators, and learn how to time Python code to measure performance here.
Python modules let you organize statements and functions into larger components, and this part illustrates how to create, use, and reload modules. We’ll also look at some more advanced topics here, such as module packages, module reloading, package-relative imports, 3.3’s new namespace packages, and the__name__variable.
Here, we explore Python’s object-oriented programming tool, the class — an optional but powerful way to structure code for customization and reuse, which almost naturally minimizes redundancy. As you’ll see, classes mostly reuse ideas we will have covered by this point in the book, and OOP in Python is mostly about looking up names in linked objects with a special first argument in functions. As you’ll also see, OOP is optional in Python, but most find Python’s OOP to be much simpler than others, and it can shave development time substantially, especially for long-term strategic project development.
We conclude the language fundamentals coverage in this text with a look at Python’s exception handling model and statements, plus a brief overview of development tools that will become more useful when you start writing larger programs (debugging and testing tools, for instance). Although exceptions are a fairly lightweight tool, this part appears after the discussion of classes because user-defined exceptions should now all be classes. We also cover some more advanced topics, such as context managers, here.
In the final part, we explore some advanced topics: Unicode and byte strings, managed attribute tools like properties and descriptors, function and class decorators, and metaclasses. These chapters are all optional reading, because not all programmers need to understand the subjects they address. On the other hand, readers who must process internationalized text or binary data, or are responsible for developing APIs for other programmers to use, should find something of interest in this part. The examples here are also larger than most of those in this book, and can serve as self-study material.
The book wraps up with a set of four appendixes that give platform-specific tips for installing and using Python on various computers; present the new Windows launcher that ships with Python 3.3; summarize changes in Python addressed by recent editions and give links to their coverage here; and provide solutions to the end-of-part exercises. Solutions to end-of-chapter quizzes appear in the chapters themselves.
As implied by the preceding structural description, you can use the index and table of contents to hunt for details, but there are no reference appendixes in this book. If you are looking for Python reference resources (and most readers probably will be very soon in their Python careers), I suggest the previously mentioned book that I also wrote as a companion to this one — Python Pocket Reference — as well as other reference books you’ll find with a quick search, and the standard Python reference manuals maintained at http://www.python.org. The latter of these are free, always up to date, and available both on the Web and on your computer after a Windows install.
As also discussed earlier, this book is not a guide to specific applications such as the Web, GUIs, or systems programming. By proxy, this includes the libraries and tools used in applications work; although some standard libraries and tools are introduced here — includingtimeit,shelve,pickle,struct,json,pdb,os,urllib,re,xml,random, PyDoc and IDLE — they are not officially in this book’s primary scope. If you’re looking for more coverage on such topics and are already proficient with Python, I recommend the follow-up book Programming Python, among others. That book assumes this one as its prerequisite, though, so be sure you have a firm grasp of the core language first. Especially in an engineering domain like software, one must walk before one runs.
format method, and some dict calls rely on function
keyword arguments.list calls used around many tools, imply
iteration concepts.exec to run code now
assumes knowledge of file objects and
interfaces.This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Learning Python, Fifth Edition, by Mark Lutz. Copyright 2013 Mark Lutz, 978-1-4493-5573-9.”If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.
Used for email addresses, URLs, filenames, pathnames, and emphasizing new terms when they are first introduced
Constant widthUsed for program code, the contents of files and the output from commands, and to designate modules, methods, statements, and system commands
Constant width
boldUsed in code sections to show commands or text that would be typed by the user, and, occasionally, to highlight portions of code
Constant width italicUsed for replaceables and some comments in code sections
This site will maintain this edition’s official list of book errata, and chronicle specific patches applied to the text in reprints. It’s also the official site for the book’s examples as described earlier.
This site will be used to post more general updates related to this text or Python itself — a hedge against future changes, which should be considered a sort of virtual appendix to this book.
Mark Lutz, Amongst the Larch, Spring 2013
1 And 2007’s short-lived third edition covered Python 2.5, and its simpler — and shorter — single-line Python world. See http://learning-python.com/books for more on this book’s history. Over the years, this book has grown in size and complexity in direct proportion to Python’s own growth. Per Appendix C, Python 3.0 alone introduced 27 additions and 57 changes in the language that found their way into this book, and Python 3.3 continues this trend. Today’s Python programmer faces two incompatible lines, three major paradigms, a plethora of advanced tools, and a blizzard of feature redundancy — most of which do not divide neatly between the 2.X and 3.X lines. That’s not as daunting as it may sound (many tools are variations on a theme), but all are fair game in an inclusive, comprehensive Python text.
2 The standard disclaimer: I wrote this and another book mentioned earlier, which work together as a set: Learning Python for language fundamentals, Programming Python for applications basics, and Python Pocket Reference as a companion to the other two. All three derive from 1995’s original and broad Programming Python. I encourage you to explore the many Python books available today (I stopped counting at 200 at Amazon.com just now because there was no end in sight, and this didn’t include related subjects like Django). My own publisher has recently produced Python-focused books on instrumentation, data mining, App Engine, numeric analysis, natural language processing, MongoDB, AWS, and more — specific domains you may wish to explore once you’ve mastered Python language fundamentals here. The Python story today is far too rich for any one book to address alone.
3 Mostly under Windows 7, but it’s irrelevant to this book. At this writing, Python installs on Windows 8 and runs in its desktop mode, which is essentially the same as Windows 7 without a Start button as I write this (you may need to create shortcuts for former Start button menu items). Support for WinRT/Metro “apps” is still pending. See Appendix A for more details. Frankly, the future of Windows 8 is unclear as I type these words, so this book will be as version-neutral as possible.
For many, Python’s focus on readability, coherence, and software quality in general sets it apart from other tools in the scripting world. Python code is designed to be readable, and hence reusable and maintainable — much more so than traditional scripting languages. The uniformity of Python code makes it easy to understand, even if you did not write it. In addition, Python has deep support for more advanced software reuse mechanisms, such as object-oriented (OO) and functional programming.
Python boosts developer productivity many times beyond compiled or statically typed languages such as C, C++, and Java. Python code is typically one-third to one-fifth the size of equivalent C++ or Java code. That means there is less to type, less to debug, and less to maintain after the fact. Python programs also run immediately, without the lengthy compile and link steps required by some other tools, further boosting programmer speed.
Most Python programs run unchanged on all major computer platforms. Porting Python code between Linux and Windows, for example, is usually just a matter of copying a script’s code between machines. Moreover, Python offers multiple options for coding portable graphical user interfaces, database access programs, web-based systems, and more. Even operating system interfaces, including program launches and directory processing, are as portable in Python as they can possibly be.
Python comes with a large collection of prebuilt and portable functionality, known as the standard library. This library supports an array of application-level programming tasks, from text pattern matching to network scripting. In addition, Python can be extended with both homegrown libraries and a vast collection of third-party application support software. Python’s third-party domain offers tools for website construction, numeric programming, serial port access, game development, and much more (see ahead for a sampling). The NumPy extension, for instance, has been described as a free and more powerful equivalent to the Matlab numeric programming system.
Python scripts can easily communicate with other parts of an application, using a variety of integration mechanisms. Such integrations allow Python to be used as a product customization and extension tool. Today, Python code can invoke C and C++ libraries, can be called from C and C++ programs, can integrate with Java and .NET components, can communicate over frameworks such as COM and Silverlight, can interface with devices over serial ports, and can interact over networks with interfaces like SOAP, XML-RPC, and CORBA. It is not a standalone tool.
Because of Python’s ease of use and built-in toolset, it can make the act of programming more pleasure than chore. Although this may be an intangible benefit, its effect on productivity is an important asset.
Sometimes when people hear Python described as a scripting language, they think it means that Python is a tool for coding operating-system-oriented scripts. Such programs are often launched from console command lines and perform tasks such as processing text files and launching other programs.Python programs can and do serve such roles, but this is just one of dozens of common Python application domains. It is not just a better shell-script language.
To others, scripting refers to a “glue” layer used to control and direct (i.e., script) other application components. Python programs are indeed often deployed in the context of larger applications. For instance, to test hardware devices, Python programs may call out to components that give low-level access to a device. Similarly, programs may run bits of Python code at strategic points to support end-user product customization without the need to ship and recompile the entire system’s source code.Python’s simplicity makes it a naturally flexible control tool. Technically, though, this is also just a common Python role; many (perhaps most) Python programmers code standalone scripts without ever using or knowing about any integrated components. It is not just a control language.
Probably the best way to think of the term “scripting language” is that it refers to a simple language used for quickly coding tasks. This is especially true when the term is applied to Python, which allows much faster program development than compiled languages like C++. Its rapid development cycle fosters an exploratory, incremental mode of programming that has to be experienced to be appreciated.Don’t be fooled, though — Python is not just for simple tasks. Rather, it makes tasks simple by its ease of use and flexibility. Python has a simple feature set, but it allows programs to scale up in sophistication as needed. Because of that, it is commonly used for quick tactical tasks and longer-term strategic development.
xml
library package, the xmlrpclib
module, and third-party extensionsjson and csv modulesPython keeps track of the kinds of objects your program uses when it runs; it doesn’t require complicated type and size declarations in your code. In fact, as you’ll see in Chapter 6, there is no such thing as a type or variable declaration anywhere in Python. Because Python code does not constrain data types, it is also usually automatically applicable to a whole range of objects.
For building larger systems, Python includes tools such as modules, classes, and exceptions. These tools allow you to organize systems into components, use OOP to reuse and customize code, and handle events and errors gracefully. Python’s functional programming tools, described earlier, provide additional ways to meet many of the same goals.
Python provides commonly used data structures such as lists, dictionaries, and strings as intrinsic parts of the language; as you’ll see, they’re both flexible and easy to use. For instance, built-in objects can grow and shrink on demand, can be arbitrarily nested to represent complex information, and more.
For more specific tasks, Python also comes with a large collection of precoded library tools that support everything from regular expression matching to networking. Once you learn the language itself, Python’s library tools are where much of the application-level action occurs.
Because Python is open source, developers are encouraged to contribute precoded tools that support tasks beyond those supported by its built-ins; on the Web, you’ll find free support for COM, imaging, numeric programming, XML, database access, and much more.
import this statement?import this triggers an Easter egg inside
Python that displays some of the design philosophies underlying the
language. You’ll learn how to run this statement in the next
chapter.1 For a more complete look at the Python philosophy, type the
command import this at any Python
interactive prompt (you’ll see how in Chapter 3). This invokes an “Easter egg”
hidden in Python — a collection of design principles underlying Python
that permeate both the language and its user community. Among them,
the acronym EIBTI is now fashionable jargon for the “explicit is
better than implicit” rule. These principles are not religion, but
are close enough to qualify as a Python motto and creed, which we’ll
be quoting from often in this book.
print('hello world')
print(2 ** 100)hello world 1267650600228229401496703205376
C:\code> python script0.py
hello world
1267650600228229401496703205376%pythonPython 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit ... Type "help", "copyright", "credits" or "license" for more information. >>>^Z
python
in a DOS console window — a program named cmd.exe and usually known as
Command Prompt. For more details on starting
this program, see this chapter’s sidebar “Where Is Command Prompt on Windows?”.python in the window that
opens up.python in a shell window, you
can also begin similar interactive sessions by starting the IDLE GUI
(discussed later), or by selecting the “Python (command line)” menu
option from the Start button menu for Python, as shown in Figure 2-1 in Chapter 2. Both spawn a Python
interactive prompt with the same functionality obtained with a
“python” command.c:\code>c:\python33\pythonPython 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit ... Type "help", "copyright", "credits" or "license" for more information. >>>^Z
c:\code>cd c:\python33c:\Python33>pythonPython 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit ... Type "help", "copyright", "credits" or "license" for more information. >>>^Z
c:\code>pyPython 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit ... Type "help", "copyright", "credits" or "license" for more information. >>>^Zc:\code>py −2Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] ... Type "help", "copyright", "credits" or "license" for more information. >>>^Zc:\code>py −3.1Python 3.1.4 (default, Jun 12 2011, 14:16:16) [MSC v.1500 64 bit (AMD64)] ... Type "help", "copyright", "credits" or "license" for more information. >>>^Z
mkdir command, usually after
you cd to your desired parent
directory (e.g., cd c:\ and
mkdir code). Your working
directory can be located wherever you like and called whatever you
wish, and doesn’t have to be C:\code (I chose this name because it’s
short in prompts). But running out of one directory will help you
keep track of your work and simplify some tasks. For more Windows
hints, see this chapter’s sidebar on Command Prompt, as well as
Appendix A.mkdir command in a shell window
or file explorer GUI specific to your platform, but the same
concepts apply. The Cygwin Unix-like system for Windows is similar
too, though your directory names may vary (/home and /cygdrive/c are candidates).%python>>>print('Hello world!')Hello world! >>>print(2 ** 8)256
>>>lumberjack = 'okay'>>>lumberjack'okay' >>>2 ** 8256 >>>^Z# Use Ctrl-D (on Unix) or Ctrl-Z (on Windows) to exit%
%python>>>'Spam!' * 8# Learning by trying'Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!'
>>>X# Making mistakesTraceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'X' is not defined
>>>import os>>>os.getcwd()# Testing on the fly'c:\\code'
os.system), but
they are not as direct as simply typing the commands
themselves.print statements are required only in files.
Because the interactive interpreter automatically prints the
results of expressions, you do not need to type complete print statements interactively. This is a
nice feature, but it tends to confuse users when they move on to
writing code in files: within a code file, you must use print statements to see your output
because expression results are not automatically echoed. Remember,
you must say print in files, but
it’s optional interactively.... instead of >>> for lines 2 and beyond; in
the IDLE GUI interface, lines after the first are instead
automatically indented.... prompt or a
blank line when entering your code, it probably means that you’ve
somehow confused interactive Python into thinking you’re typing a
multiline statement. Try hitting the Enter key or a Ctrl-C
combination to get back to the main prompt. The >>> and ... prompt strings can also be changed
(they are available in the built-in module sys), but I’ll assume they have not been
in the book’s example listings.>>>for x in 'spam':...print(x)# Press Enter twice here to make this loop run...
>>>for x in 'spam':...print(x)# Press Enter twice before a new statement...print('done')File "<stdin>", line 3 print('done') ^ SyntaxError: invalid syntax
# A first Python scriptimport sys# Load a library moduleprint(sys.platform) print(2 ** 100)# Raise 2 to a powerx = 'Spam!' print(x * 8)# String repetition
print function
calls, to display the script’s resultsx,
created when it’s assigned, to hold onto a string object% python script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!% python script1.py > saveit.txtC:\code> python script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!C:\code> C:\python33\python script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!c:\code> py −3 script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!C:\code> script1.pyC:\code>cd D:\otherD:\other>python c:\code\script1.py
D:\other> C:\Python33\python c:\code\script1.pypython
script1.py rather than python
script1. By contrast, Python’s import statements, which we’ll meet later
in this chapter, omit both the .py file suffix and the directory path
(e.g., import script1). This may
seem trivial, but confusing these two is a common mistake.python
d:\tests\spam.py). Within Python code, however, you can
just say import spam and rely on
the Python module search path to locate your file, as described
later.print statements
in files. Yes, we’ve already been over this, but it is
such a common mistake that it’s worth repeating at least once here.
Unlike in interactive coding, you generally must use print statements to see output from
program files. If you don’t see any output, make sure you’ve said
“print” in your file. print
statements are not required in an interactive
session, since Python automatically echoes expression results;
prints don’t hurt here, but are
superfluous typing.#!
(often called “hash bang” or “shebang”), followed by the path to the
Python interpreter on your machine.chmod +x
file.py usually does the trick.#!/usr/local/bin/python
print('The Bright Side ' + 'of Life...') # + means concatenate for strings% brian
The Bright Side of Life...#!/usr/bin/env python
...script goes here...C:\code> python brian
The Bright Side of Life...c:\code>type robin3.py#!/usr/bin/python3 print('Run', 'away!...')# 3.X functionc:\code>py robin3.py# Run file per #! line versionRun away!... c:\code>type robin2.py#!python2 print 'Run', 'away more!...'# 2.X statementc:\code>py robin2.py# Run file per #! line versionRun away more!...
c:\code>py −3.1 robin3.py# Run per command-line argumentRun away!...
On Windows, the Registry makes opening files with icon clicks easy. When installed, Python uses Windows filename associations to automatically register itself to be the program that opens Python program files when they are clicked. Because of that, it is possible to launch the Python programs you write by simply clicking (or double-clicking) on their file icons with your mouse cursor.Specifically, a clicked file will be run by one of two Python programs, depending on its extension and the Python you’re running. In Pythons 3.2 and earlier, .py files are run bypython.exewith a console (Command Prompt) window, and .pyw files are run bypythonw.exewithout a console. Byte code files are also run by these programs if clicked. Per Appendix B, in Python 3.3 and later (and where it’s installed separately), the new Window’s launchers’spy.exeandpyw.exeprograms serve the same roles, opening .py and .pyw files, respectively.
On non-Windows systems, you will probably be able to perform a similar feat, but the icons, file explorer navigation schemes, and more may differ slightly. On Mac OS X, for instance, you might use PythonLauncher in the MacPython (or Python N.M) folder of your Applications folder to run by clicking in Finder.On some Linux and other Unix systems, you may need to register the .py extension with your file explorer GUI, make your script executable using the#!line scheme of the preceding section, or associate the file MIME type with an application or command by editing files, installing programs, or using other tools. See your file explorer’s documentation for more details.
# A first Python scriptimport sys# Load a library moduleprint(sys.platform) print(2 ** 100)# Raise 2 to a powerx = 'Spam!' print(x * 8)# String repetition
C:\code> python script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!# A first Python scriptimport sys# Load a library moduleprint(sys.platform) print(2 ** 100)# Raise 2 to a powerx = 'Spam!' print(x * 8)# String repetitioninput()# <== ADDED
input('Press Enter to
exit'))nextinput = input())python spam.py <
input.txt), just as the print statement does for outputC:\code>C:\python33\python>>>import script1win32 1267650600228229401496703205376 Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!
...Change script1.py in a text edit window to print 2 ** 16...>>>import script1>>>import script1
>>>from imp import reload# Must load from module in 3.X (only)>>>reload(script1)win32 65536 Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam! <module 'script1' from '.\\script1.py'> >>>
title = "The Meaning of Life"
%python# Start Python>>>import myfile# Run file; load module as a whole>>>myfile.title# Use its attribute names: '.' to qualify'The Meaning of Life'
%python# Start Python>>>from myfile import title# Run file; copy its names>>>title# Use name directly: no need to qualify'The Meaning of Life'
a = 'dead'# Define three attributesb = 'parrot'# Exported to other filesc = 'sketch' print(a, b, c)# Also used in this file (in 2.X: print a, b, c)
% python threenames.py
dead parrot sketch%python>>>import threenames# Grab the whole module: it runs heredead parrot sketch >>> >>>threenames.b, threenames.c# Access its attributes('parrot', 'sketch') >>> >>>from threenames import a, b, c# Copy multiple names out>>>b, c('parrot', 'sketch')
>>> dir(threenames)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'a', 'b', 'c']%python>>>exec(open('script1.py').read())win32 65536 Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!...Change script1.py in a text edit window to print 2 ** 32...>>>exec(open('script1.py').read())win32 4294967296 Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!
>>>x = 999>>>exec(open('script1.py').read())# Code run in this namespace by default...same output...>>>x# Its assignments can overwrite names here'Spam!'
find for the exact
location).c:\code>python -m idlelib.idle# Run idle.py in a package on module path
quit
function in your code, the normal way to exit a GUI program, may be
enough to cause your program’s GUI to hang if run in IDLE (destroy may be better here only). Your
code may not exhibit such problems, but as a rule of thumb, it’s
always safe to use IDLE to edit GUI programs but launch them using
other options, such as icon clicks or system command lines. When in
doubt, if your code fails in IDLE, try it outside the GUI.-n command-line flag forces this mode. On
Windows, for example, start a Command Prompt window and run the
system command line idle.py
-n from within the directory C:\Python33\Lib\idlelib (cd there first if needed). A python -m idlelib.idle –n command works
from anywhere (see Appendix A
for –m).import commands to access names at the top
level of files you’ve already run. This can be handy, but it can
also be confusing, because outside the IDLE environment names must
always be imported from files explicitly to be used.Eclipse is an advanced open source IDE GUI. Originally developed as a Java IDE, Eclipse also supports Python development when you install the PyDev (or a similar) plug-in. Eclipse is a popular and powerful option for Python development, and it goes well beyond IDLE’s feature set. It includes support for code completion, syntax highlighting, syntax analysis, refactoring, debugging, and more. Its downsides are that it is a large system to install and may require shareware extensions for some features (this may vary over time). Still, when you are ready to graduate from IDLE, the Eclipse/PyDev combination is worth your attention.
A full-featured development environment GUI for Python (and other languages), Komodo includes standard syntax coloring, text editing, debugging, and other features. In addition, Komodo offers many advanced features that IDLE does not, including project files, source-control integration, and regular-expression debugging. At this writing, Komodo is not free, but see the Web for its current status — it is available at http://www.activestate.com from ActiveState, which also offers the ActivePython distribution package mentioned in Appendix A.
NetBeans is a powerful open source development environment GUI with support for many advanced features for Python developers: code completion, automatic indentation and code colorization, editor hints, code folding, refactoring, debugging, code coverage and testing, projects, and more. It may be used to develop both CPython and Jython code. Like Eclipse, NetBeans requires installation steps beyond those of the included IDLE GUI, but it is seen by many as more than worth the effort. Search the Web for the latest information and links.
PythonWin is a free Windows-only IDE for Python that ships as part of ActiveState’s ActivePython distribution (and may also be fetched separately from http://www.python.org resources). It is roughly like IDLE, with a handful of useful Windows-specific extensions added; for example, PythonWin has support for COM objects. Today, IDLE is probably more advanced than PythonWin (for instance, IDLE’s dual-process architecture often prevents it from hanging). However, PythonWin still offers tools for Windows developers that IDLE does not. See http://www.activestate.com for more information.
Other IDEs are popular among Python developers too, including the mostly commercial Wing IDE, Microsoft Visual Studio via a plug-in, and PyCharm, PyScripter, Pyshield, and Spyder — but I do not have space to do justice to them here, and more will undoubtedly appear over time. In fact, almost every programmer-friendly text editor has some sort of support for Python development these days, whether it be preinstalled or fetched separately. Emacs and Vim, for instance, have substantial Python support.IDE choices are often subjective, so I encourage you to browse to find tools that fit your development style and goals. For more information, see the resources available at http://www.python.org or search the Web for “Python IDE” or similar. A search for “Python editors” today leads you to a wiki page that maintains information about dozens of IDE and text-editor options for Python programming.
#include <Python.h>
...
Py_Initialize(); // This is C, not Python
PyRun_SimpleString("x = 'brave ' + 'sir robin'"); // But it runs Python codepython as a
system command line in your system’s console window (a Command Prompt
window on Windows). Another alternative is to launch IDLE, as its main
Python shell window is an interactive session. Depending on your
platform and Python, if you have not set your system’s PATH variable to find Python, you may need
to cd to where Python is installed,
or type its full directory path instead of just python (e.g., C:\Python33\python on Windows, unless
you’re using the 3.3 launcher).exec built-in function, and IDE GUI
selections such as IDLE’s Run→Run Module menu option. On Unix, they
can also be run as executables with the #! trick, and some platforms support more
specialized launching techniques (e.g., drag and drop). In addition,
some text editors have unique ways to run Python code, some Python
programs are provided as standalone “frozen binary” executables, and
some systems use Python code in embedded mode, where it is run
automatically by an enclosing program written in a language like C,
C++, or Java. The latter technique is usually done to provide a user
customization layer.input trick comes in handy);
error messages generated by your script also appear in an output
window that closes before you can examine its contents (which is one
reason that system command lines and IDEs such as IDLE are better for
most development).>>> prompt), and type the
expression "Hello World!"
(including the quotes). The string should be echoed back to you. The
purpose of this exercise is to get your environment configured to run
Python. In some scenarios, you may need to first run a cd shell command, type the full path to the
Python executable, or add its path to your PATH environment variable. If desired, you
can set PATH in your .cshrc or .kshrc file to make Python permanently
available on Unix systems; on Windows, the environment variable GUI is
usually what you want for this. See Appendix A for help with environment
variable settings.print('Hello module world!') and
store it as module1.py. Now, run
this file by using any launch option you like: running it in IDLE,
clicking on its file icon, passing it to the Python interpreter on the
system shell’s command line (e.g., python
module1.py), built-in exec calls, imports and reloads, and so on.
In fact, experiment by running your file with as many of the launch
techniques discussed in this chapter as you can. Which technique seems
easiest? (There is no right answer to this, of course.)>>> prompt)
and import the module you wrote in exercise 2. Try moving the file to
a different directory and importing it again from its original
directory (i.e., run Python in the original directory when you
import). What happens? (Hint: is there still a module1.pyc byte code file in the original
directory, or something similar in a __pycache__ subdirectory there?)#! line to the top of your
module1.py module file, give the
file executable privileges, and run it directly as an executable. What
does the first line need to contain? #! usually only has meaning on Unix, Linux,
and Unix-like platforms such as Mac OS X; if you’re working on
Windows, instead try running your file by listing just its name in a
Command Prompt window without the word “python” before it (this works
on recent versions of Windows), via the Start→Run... dialog box, or
similar. If you are using Python 3.3 or the Windows launcher that
installs with it, experiment with changing your script’s #! line to launch different Python versions
you may have installed on your computer (or equivalently, work through
the tutorial in Appendix B).2 ** 500 and 1 / 0, and reference an undefined variable
name as we did early on in this chapter. What happens?L = [1, 2]# Make a 2-item listL.append(L)# Append L as a single item to itselfL# Print L: a cyclic/circular object
help function) in Chapter 15. If you still have time, go
explore the Python website, as well as its PyPI third-party extension
repository. Especially check out the Python.org (http://www.python.org) documentation and search pages;
they can be crucial resources.1 As we discussed when exploring command lines, all recent
Windows versions also let you type just the name of a .py file at the system command line — they
use the Registry to determine that the file should be opened with
Python (e.g., typing brian.py
is equivalent to typing python
brian.py). This command-line mode is similar in spirit
to the Unix #!, though it is
system-wide on Windows, not per-file. It also requires an explicit
.py extension: filename
associations won’t work without it. Some
programs may actually interpret and use a first
#! line on Windows much like on
Unix (including Python 3.3’s Windows launcher), but the system shell
on Windows itself simply ignores it.
2 Conversely, it is also possible to completely suppress the pop-up console window (a.k.a. Command Prompt) for clicked files on Windows when you don’t want to see printed text. Files whose names end in a .pyw extension will display only windows constructed by your script, not the default console window. .pyw files are simply .py source files that have this special operational behavior on Windows. They are mostly used for Python-coded user interfaces that build windows of their own, often in conjunction with various techniques for saving printed output and errors to files. As implied earlier, Python achieves this when it is installed by associating a special executable (pythonw.exe in 3.2 and earlier and pyw.exe as of 3.3) to open .pyw files when clicked.
3 Notice that import and
from both list the name of the
module file as simply myfile
without its .py extension
suffix. As you’ll learn in Part V,
when Python looks for the actual file, it knows to include the
suffix in its search procedure. Again, you must include the .py suffix in system shell command lines,
but not in import
statements.
4 If you’re too curious to wait, the short story is that Python
searches for imported modules in every directory listed in sys.path — a Python list of directory name
strings in the sys module, which
is initialized from a PYTHONPATH
environment variable, plus a set of standard directories. If you
want to import from a directory other than the one you are working
in, that directory must generally be listed in your PYTHONPATH setting. For more details, see
Chapter 22 and Appendix A.
5 IDLE is officially a corruption of IDE, but it’s really named in honor of Monty Python member Eric Idle. See Chapter 1 if you’re not sure why.
6 See Programming Python (O’Reilly) for more details on embedding Python in C/C++. The embedding API can call Python functions directly, load modules, and more. Also, note that the Jython system allows Java programs to invoke Python code using a Java-based API (a Python interpreter class).
| Object type | Example literals/creation |
|---|---|
>>> 'spam'>>>123 + 222# Integer addition345 >>>1.5 * 4# Floating-point multiplication6.0 >>>2 ** 100# 2 to the power 100, again1267650600228229401496703205376
>>>len(str(2 ** 1000000))# How many digits in a really BIG number?301030
>>>3.1415 * 2# repr: as code (Pythons < 2.7 and 3.1)6.2830000000000004 >>>print(3.1415 * 2)# str: user-friendly6.283
>>>3.1415 * 2# repr: as code (Pythons >= 2.7 and 3.1)6.283
>>>import math>>>math.pi3.141592653589793 >>>math.sqrt(85)9.219544457292887
>>>import random>>>random.random()0.7082048489415967 >>>random.choice([1, 2, 3, 4])1
>>>S = 'Spam'# Make a 4-character string, and assign it to a name>>>len(S)# Length4 >>>S[0]# The first item in S, indexing by zero-based position'S' >>>S[1]# The second item from the left'p'
>>>S[-1]# The last item from the end in S'm' >>>S[-2]# The second-to-last item from the end'a'
>>>S[-1]# The last item in S'm' >>>S[len(S)-1]# Negative indexing, the hard way'm'
>>>S# A 4-character string'Spam' >>>S[1:3]# Slice of S from offsets 1 through 2 (not 3)'pa'
>>>S[1:]# Everything past the first (1:len(S))'pam' >>>S# S itself hasn't changed'Spam' >>>S[0:3]# Everything but the last'Spa' >>>S[:3]# Same as S[0:3]'Spa' >>>S[:-1]# Everything but the last again, but simpler (0:-1)'Spa' >>>S[:]# All of S as a top-level copy (0:len(S))'Spam'
>>>S'Spam' >>>S + 'xyz'# Concatenation'Spamxyz' >>>S# S is unchanged'Spam' >>>S * 8# Repetition'SpamSpamSpamSpamSpamSpamSpamSpam'
>>>S'Spam' >>>S[0] = 'z'# Immutable objects cannot be changed...error text omitted...TypeError: 'str' object does not support item assignment >>>S = 'z' + S[1:]# But we can run expressions to make new objects>>>S'zpam'
>>>S = 'shrubbery'>>>L = list(S)# Expand to a list: [...]>>>L['s', 'h', 'r', 'u', 'b', 'b', 'e', 'r', 'y'] >>>L[1] = 'c'# Change it in place>>>''.join(L)# Join with empty delimiter'scrubbery' >>>B = bytearray(b'spam')# A bytes/list hybrid (ahead)>>>B.extend(b'eggs')# 'b' needed in 3.X, not 2.X>>>B# B[i] = ord(x) works here toobytearray(b'spameggs') >>>B.decode()# Translate to normal string'spameggs'
>>>S = 'Spam'>>>S.find('pa')# Find the offset of a substring in S1 >>>S'Spam' >>>S.replace('pa', 'XYZ')# Replace occurrences of a string in S with another'SXYZm' >>>S'Spam'
>>>line = 'aaa,bbb,ccccc,dd'>>>line.split(',')# Split on a delimiter into a list of substrings['aaa', 'bbb', 'ccccc', 'dd'] >>>S = 'spam'>>>S.upper()# Upper- and lowercase conversions'SPAM' >>>S.isalpha()# Content tests: isalpha, isdigit, etc.True >>>line = 'aaa,bbb,ccccc,dd\n'>>>line.rstrip()# Remove whitespace characters on the right side'aaa,bbb,ccccc,dd' >>>line.rstrip().split(',')# Combine two operations['aaa', 'bbb', 'ccccc', 'dd']
>>>'%s, eggs, and %s' % ('spam', 'SPAM!')# Formatting expression (all)'spam, eggs, and SPAM!' >>>'{0}, eggs, and {1}'.format('spam', 'SPAM!')# Formatting method (2.6+, 3.0+)'spam, eggs, and SPAM!' >>>'{}, eggs, and {}'.format('spam', 'SPAM!')# Numbers optional (2.7+, 3.1+)'spam, eggs, and SPAM!'
>>>'{:,.2f}'.format(296999.2567)# Separators, decimal digits'296,999.26' >>>'%.2f | %+05d' % (3.14159, −42)# Digits, padding, signs'3.14 | −0042'
>>> dir(S)
['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__',
'__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__',
'__getnewargs__', '__gt__', '__hash__', '__init__', '__iter__', '__le__',
'__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__',
'__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count',
'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index',
'isalnum', 'isalpha', 'isdecimal', 'isdigit', 'isidentifier', 'islower',
'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust',
'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex',
'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith',
'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']>>>S + 'NI!''spamNI!' >>>S.__add__('NI!')'spamNI!'
>>> help(S.replace)
Help on built-in function replace:
replace(...)
S.replace(old, new[, count]) -> str
Return a copy of S with all occurrences of substring
old replaced by new. If the optional argument count is
given, only the first count occurrences are replaced.>>>S = 'A\nB\tC'# \n is end-of-line, \t is tab>>>len(S)# Each stands for just one character5 >>>ord('\n')# \n is one character coded as decimal value 1010 >>>S = 'A\0B\0C'# \0, a binary zero byte, does not terminate string>>>len(S)5 >>>S# Non-printables are displayed as \xNN hex escapes'A\x00B\x00C'
>>>msg = """aaaaaaaaaaaaabbb'''bbbbbbbbbb""bbbbbbb'bbbbcccccccccccccc""">>>msg'\naaaaaaaaaaaaa\nbbb\'\'\'bbbbbbbbbb""bbbbbbb\'bbbb\ncccccccccccccc\n'
>>>'sp\xc4m'# 3.X: normal str strings are Unicode text'spÄm' >>>b'a\x01c'# bytes strings are byte-based datab'a\x01c' >>>u'sp\u00c4m'# The 2.X Unicode literal works in 3.3+: just str'spÄm'
>>>print u'sp\xc4m'# 2.X: Unicode strings are a distinct typespÄm >>>'a\x01c'# Normal str strings contain byte-based text/data'a\x01c' >>>b'a\x01c'# The 3.X bytes literal works in 2.6+: just str'a\x01c'
>>>'spam'# Characters may be 1, 2, or 4 bytes in memory'spam' >>>'spam'.encode('utf8')# Encoded to 4 bytes in UTF-8 in filesb'spam' >>>'spam'.encode('utf16')# But encoded to 10 bytes in UTF-16b'\xff\xfes\x00p\x00a\x00m\x00'
>>> 'sp\xc4\u00c4\U000000c4m'
'spÄÄÄm'>>> '\u00A3', '\u00A3'.encode('latin1'), b'\xA3'.decode('latin1')
('£', b'\xa3', '£')u'x' + b'y'# Works in 2.X (where b is optional and ignored)u'x' + 'y'# Works in 2.X: u'xy'u'x' + b'y'# Fails in 3.3 (where u is optional and ignored)u'x' + 'y'# Works in 3.3: 'xy''x' + b'y'.decode()# Works in 3.X if decode bytes to str: 'xy''x'.encode() + b'y'# Works in 3.X if encode str to bytes: b'xy'
>>>import re>>>match = re.match('Hello[ \t]*(.*)world', 'Hello Python world')>>>match.group(1)'Python '
>>>match = re.match('[/:](.*)[/:](.*)[/:](.*)', '/usr/home:lumberjack')>>>match.groups()('usr', 'home', 'lumberjack') >>>re.split('[/:]', '/usr/home/lumberjack')['', 'usr', 'home', 'lumberjack']
>>>L = [123, 'spam', 1.23]# A list of three different-type objects>>>len(L)# Number of items in the list3
>>>L[0]# Indexing by position123 >>>L[:-1]# Slicing a list returns a new list[123, 'spam'] >>>L + [4, 5, 6]# Concat/repeat make new lists too[123, 'spam', 1.23, 4, 5, 6] >>>L * 2[123, 'spam', 1.23, 123, 'spam', 1.23] >>>L# We're not changing the original list[123, 'spam', 1.23]
>>>L.append('NI')# Growing: add object at end of list>>>L[123, 'spam', 1.23, 'NI'] >>>L.pop(2)# Shrinking: delete an item in the middle1.23 >>>L# "del L[2]" deletes from a list too[123, 'spam', 'NI']
>>>M = ['bb', 'aa', 'cc']>>>M.sort()>>>M['aa', 'bb', 'cc'] >>>M.reverse()>>>M['cc', 'bb', 'aa']
>>>L[123, 'spam', 'NI'] >>>L[99]...error text omitted...IndexError: list index out of range >>>L[99] = 1...error text omitted...IndexError: list assignment index out of range
>>>M = [[1, 2, 3],# A 3 × 3 matrix, as nested lists[4, 5, 6],# Code can span lines if bracketed[7, 8, 9]]>>>M[[1, 2, 3], [4, 5, 6], [7, 8, 9]]
>>>M[1]# Get row 2[4, 5, 6] >>>M[1][2]# Get row 2, then get item 3 within the row6
>>>col2 = [row[1] for row in M]# Collect the items in column 2>>>col2[2, 5, 8] >>>M# The matrix is unchanged[[1, 2, 3], [4, 5, 6], [7, 8, 9]]
>>>[row[1] + 1 for row in M]# Add 1 to each item in column 2[3, 6, 9] >>>[row[1] for row in M if row[1] % 2 == 0]# Filter out odd items[2, 8]
>>>diag = [M[i][i] for i in [0, 1, 2]]# Collect a diagonal from matrix>>>diag[1, 5, 9] >>>doubles = [c * 2 for c in 'spam']# Repeat characters in a string>>>doubles['ss', 'pp', 'aa', 'mm']
>>>list(range(4))# 0..3 (list() required in 3.X) [0, 1, 2, 3] >>>list(range(−6, 7, 2))# −6 to +6 by 2 (need list() in 3.X) [−6, −4, −2, 0, 2, 4, 6] >>>[[x ** 2, x ** 3] for x in range(4)]# Multiple values, "if" filters[[0, 0], [1, 1], [4, 8], [9, 27]] >>>[[x, x / 2, x * 2] for x in range(−6, 7, 2) if x > 0][[2, 1, 4], [4, 2, 8], [6, 3, 12]]
>>>G = (sum(row) for row in M)# Create a generator of row sums>>>next(G)# iter(G) not required here6 >>>next(G)# Run the iteration protocol next()15 >>>next(G)24
>>>list(map(sum, M))# Map sum over items in M[6, 15, 24]
>>>{sum(row) for row in M}# Create a set of row sums{24, 6, 15} >>>{i : sum(M[i]) for i in range(3)}# Creates key/value table of row sums{0: 6, 1: 15, 2: 24}
>>>[ord(x) for x in 'spaam']# List of character ordinals[115, 112, 97, 97, 109] >>>{ord(x) for x in 'spaam'}# Sets remove duplicates{112, 97, 115, 109} >>>{x: ord(x) for x in 'spaam'}# Dictionary keys are unique{'p': 112, 'a': 97, 's': 115, 'm': 109} >>>(ord(x) for x in 'spaam')# Generator of values<generator object <genexpr> at 0x000000000254DAB0>
>>> D = {'food': 'Spam', 'quantity': 4, 'color': 'pink'}>>>D['food']# Fetch value of key 'food''Spam' >>>D['quantity'] += 1# Add 1 to 'quantity' value>>>D{'color': 'pink', 'food': 'Spam', 'quantity': 5}
>>>D = {}>>>D['name'] = 'Bob'# Create keys by assignment>>>D['job'] = 'dev'>>>D['age'] = 40>>>D{'age': 40, 'job': 'dev', 'name': 'Bob'} >>>print(D['name'])Bob
>>>bob1 = dict(name='Bob', job='dev', age=40)# Keywords>>>bob1{'age': 40, 'name': 'Bob', 'job': 'dev'} >>>bob2 = dict(zip(['name', 'job', 'age'], ['Bob', 'dev', 40]))# Zipping>>>bob2{'job': 'dev', 'name': 'Bob', 'age': 40}
>>>rec = {'name': {'first': 'Bob', 'last': 'Smith'},'jobs': ['dev', 'mgr'],'age': 40.5}
>>>rec['name']# 'name' is a nested dictionary{'last': 'Smith', 'first': 'Bob'} >>>rec['name']['last']# Index the nested dictionary'Smith' >>>rec['jobs']# 'jobs' is a nested list['dev', 'mgr'] >>>rec['jobs'][-1]# Index the nested list'mgr' >>>rec['jobs'].append('janitor')# Expand Bob's job description in place>>>rec{'age': 40.5, 'jobs': ['dev', 'mgr', 'janitor'], 'name': {'last': 'Smith', 'first': 'Bob'}}
>>>rec = 0# Now the object's space is reclaimed
>>>D = {'a': 1, 'b': 2, 'c': 3}>>>D{'a': 1, 'c': 3, 'b': 2} >>>D['e'] = 99# Assigning new keys grows dictionaries>>>D{'a': 1, 'c': 3, 'b': 2, 'e': 99} >>>D['f']# Referencing a nonexistent key is an error...error text omitted...KeyError: 'f'
>>>'f' in DFalse >>>if not 'f' in D:# Python's sole selection statementprint('missing')missing
>>>if not 'f' in D:print('missing')print('no, really...')# Statement blocks are indentedmissing no, really...
>>>value = D.get('x', 0)# Index but with a default>>>value0 >>>value = D['x'] if 'x' in D else 0# if/else expression form>>>value0
>>>D = {'a': 1, 'b': 2, 'c': 3}>>>D{'a': 1, 'c': 3, 'b': 2}
>>>Ks = list(D.keys())# Unordered keys list>>>Ks# A list in 2.X, "view" in 3.X: use list()['a', 'c', 'b'] >>>Ks.sort()# Sorted keys list>>>Ks['a', 'b', 'c'] >>>for key in Ks:# Iterate though sorted keysprint(key, '=>', D[key])# <== press Enter twice here (3.X print)a => 1 b => 2 c => 3
>>>D{'a': 1, 'c': 3, 'b': 2} >>>for key in sorted(D):print(key, '=>', D[key])a => 1 b => 2 c => 3
>>>for c in 'spam':print(c.upper())S P A M
>>>x = 4>>>while x > 0:print('spam!' * x)x -= 1spam!spam!spam!spam! spam!spam!spam! spam!spam! spam!
>>>squares = [x ** 2 for x in [1, 2, 3, 4, 5]]>>>squares[1, 4, 9, 16, 25]
>>>squares = []>>>for x in [1, 2, 3, 4, 5]:# This is what a list comprehension doessquares.append(x ** 2)# Both run the iteration protocol internally>>>squares[1, 4, 9, 16, 25]
>>>T = (1, 2, 3, 4)# A 4-item tuple>>>len(T)# Length4 >>T + (5, 6)# Concatenation(1, 2, 3, 4, 5, 6) >>>T[0]# Indexing, slicing, and more1
>>>T.index(4)# Tuple methods: 4 appears at offset 33 >>>T.count(4)# 4 appears once1
>>>T[0] = 2# Tuples are immutable...error text omitted...TypeError: 'tuple' object does not support item assignment >>>T = (2,) + T[1:]# Make a new tuple for a new value>>>T(2, 2, 3, 4)
>>>T = 'spam', 3.0, [11, 22, 33]>>>T[1]3.0 >>>T[2][1]22 >>>T.append(4)AttributeError: 'tuple' object has no attribute 'append'
>>>f = open('data.txt', 'w')# Make a new file in output mode ('w' is write)>>>f.write('Hello\n')# Write strings of characters to it6>>>f.write('world\n')# Return number of items written in Python 3.X6 >>>f.close()# Close to flush output buffers to disk
>>>f = open('data.txt')# 'r' (read) is the default processing mode>>>text = f.read()# Read entire file into a string>>>text'Hello\nworld\n' >>>print(text)# print interprets control charactersHello world >>>text.split()# File content is always a string['Hello', 'world']
>>> for line in open('data.txt'): print(line)>>>dir(f)[...many names omitted...'buffer', 'close', 'closed', 'detach', 'encoding', 'errors', 'fileno', 'flush', 'isatty', 'line_buffering', 'mode', 'name', 'newlines', 'read', 'readable', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines'] >>>help(f.seek)...try it and see...
>>>import struct>>>packed = struct.pack('>i4sh', 7, b'spam', 8)# Create packed binary data>>>packed# 10 bytes, not objects or textb'\x00\x00\x00\x07spam\x00\x08' >>> >>>file = open('data.bin', 'wb')# Open binary output file>>>file.write(packed)# Write packed binary data10 >>>file.close()
>>>data = open('data.bin', 'rb').read()# Open/read binary data file>>>data# 10 bytes, unalteredb'\x00\x00\x00\x07spam\x00\x08' >>>data[4:8]# Slice bytes in the middleb'spam' >>>list(data)# A sequence of 8-bit bytes[0, 0, 0, 7, 115, 112, 97, 109, 0, 8] >>>struct.unpack('>i4sh', data)# Unpack into objects again(7, b'spam', 8)
>>>S ='sp\xc4m'# Non-ASCII Unicode text>>>S'spÄm' >>>S[2]# Sequence of characters'Ä' >>>file = open('unidata.txt', 'w', encoding='utf-8')# Write/encode UTF-8 text>>>file.write(S)# 4 characters written4 >>>file.close()>>>text = open('unidata.txt', encoding='utf-8').read()# Read/decode UTF-8 text>>>text'spÄm' >>>len(text)# 4 chars (code points)4
>>>raw = open('unidata.txt', 'rb').read()# Read raw encoded bytes>>>rawb'sp\xc3\x84m' >>>len(raw)# Really 5 bytes in UTF-85
>>>text.encode('utf-8')# Manual encode to bytesb'sp\xc3\x84m' >>>raw.decode('utf-8')# Manual decode to str'spÄm'
>>>text.encode('latin-1')# Bytes differ in othersb'sp\xc4m' >>>text.encode('utf-16')b'\xff\xfes\x00p\x00\xc4\x00m\x00' >>>len(text.encode('latin-1')), len(text.encode('utf-16'))(4, 10) >>>b'\xff\xfes\x00p\x00\xc4\x00m\x00'.decode('utf-16')# But same string decoded'spÄm'
>>>import codecs>>>codecs.open('unidata.txt', encoding='utf8').read()# 2.X: read/decode textu'sp\xc4m' >>>open('unidata.txt', 'rb').read()# 2.X: read raw bytes'sp\xc3\x84m' >>>open('unidata.txt').read()# 2.X: raw/undecoded too'sp\xc3\x84m'
>>>X = set('spam')# Make a set out of a sequence in 2.X and 3.X>>>Y = {'h', 'a', 'm'}# Make a set with set literals in 3.X and 2.7>>>X, Y# A tuple of two sets without parentheses({'m', 'a', 'p', 's'}, {'m', 'a', 'h'}) >>>X & Y# Intersection{'m', 'a'} >>>X | Y# Union{'m', 'h', 'a', 'p', 's'} >>>X - Y# Difference{'p', 's'} >>>X > Y# SupersetFalse >>>{n ** 2 for n in [1, 2, 3, 4]}# Set comprehensions in 3.X and 2.7{16, 1, 4, 9}
>>>list(set([1, 2, 1, 3, 1]))# Filtering out duplicates (possibly reordered)[1, 2, 3] >>>set('spam') - set('ham')# Finding differences in collections{'p', 's'} >>>set('spam') == set('asmp')# Order-neutral equality ('spam'=='asmp' False)True
>>> 'p' in set('spam'), 'p' in 'spam', 'ham' in ['eggs', 'spam', 'ham']
(True, True, True)>>>1 / 3# Floating-point (add a .0 in Python 2.X)0.3333333333333333 >>>(2/3) + (1/2)1.1666666666666665 >>>import decimal# Decimals: fixed precision>>>d = decimal.Decimal('3.141')>>>d + 1Decimal('4.141') >>>decimal.getcontext().prec = 2>>>decimal.Decimal('1.00') / decimal.Decimal('3.00')Decimal('0.33') >>>from fractions import Fraction# Fractions: numerator+denominator>>>f = Fraction(2, 3)>>>f + 1Fraction(5, 3) >>>f + Fraction(1, 2)Fraction(7, 6)
>>>1 > 2, 1 < 2# Booleans(False, True) >>>bool('spam')# Object's Boolean valueTrue >>>X = None# None placeholder>>>print(X)None >>>L = [None] * 100# Initialize a list of 100 Nones>>>L[None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None,...a list of 100 Nones...]
# In Python 2.X:>>>type(L)# Types: type of L is list type object<type 'list'> >>>type(type(L))# Even types are objects<type 'type'># In Python 3.X:>>>type(L)# 3.X: types are classes, and vice versa<class 'list'> >>>type(type(L))# See Chapter 32 for more on class types<class 'type'>
>>>if type(L) == type([]):# Type testing, if you must...print('yes')yes >>>if type(L) == list:# Using the type nameprint('yes')yes >>>if isinstance(L, list):# Object-oriented testsprint('yes')yes
>>>class Worker:def __init__(self, name, pay):# Initialize when createdself.name = name# self is the new objectself.pay = paydef lastName(self):return self.name.split()[-1]# Split string on blanksdef giveRaise(self, percent):self.pay *= (1.0 + percent)# Update pay in place
>>>bob = Worker('Bob Smith', 50000)# Make two instances>>>sue = Worker('Sue Jones', 60000)# Each has name and pay attrs>>>bob.lastName()# Call method: bob is self'Smith' >>>sue.lastName()# sue is the self subject'Jones' >>>sue.giveRaise(.10)# Updates sue's pay>>>sue.pay66000.0
None, and Booleans are sometimes
classified this way as well. There are multiple number types (integer,
floating point, complex, fraction, and decimal) and multiple string
types (simple strings and Unicode strings in Python 2.X, and text
strings and byte strings in Python 3.X).'spam', for example, is an
expression that makes a string and determines the set of operations
that can be applied to it. Because of this, core types are hardwired
into Python’s syntax. In contrast, you must call the built-in open function to create a file object (even
though this is usually considered a core type too).+) depends on the objects being
operated on. This turns out to be a key idea (perhaps
the key idea) behind using Python well — not
constraining code to specific types makes that code automatically
applicable to many types.1 Pardon my formality. I’m a computer scientist.
2 In this book, the term literal simply means
an expression whose syntax generates an object — sometimes also called a
constant. Note that the term “constant” does not
imply objects or variables that can never be changed (i.e., this term
is unrelated to C++’s const or
Python’s “immutable” — a topic explored in the section “Immutability”).
3 This matrix structure works for small-scale tasks, but for more serious number crunching you will probably want to use one of the numeric extensions to Python, such as the open source NumPy and SciPy systems. Such tools can store and process large matrixes much more efficiently than our nested list structure. NumPy has been said to turn Python into the equivalent of a free and more powerful version of the Matlab system, and organizations such as NASA, Los Alamos, JPL, and many others use this tool for scientific and financial tasks. Search the Web for more details.
4 Two application notes here. First, as a preview, the rec record we just created really could be
an actual database record, when we employ Python’s object
persistence system — an easy way to store native Python
objects in simple files or access-by-key databases, which translates
objects to and from serial byte streams automatically. We won’t go
into details here, but watch for coverage of Python’s pickle and shelve persistence modules in Chapter 9, Chapter 28, Chapter 31, and Chapter 37, where we’ll explore them in
the context of files, an OOP use case, classes, and 3.X changes,
respectively.
Second, if you are familiar with JSON
(JavaScript Object Notation) — an emerging data-interchange format
used for databases and network transfers — this example may also look
curiously similar, though Python’s support for variables, arbitrary
expressions, and changes can make its data structures more general.
Python’s json library module
supports creating and parsing JSON text, but the translation to
Python objects is often trivial. Watch for a JSON example that uses
this record in Chapter 9 when we study
files. For a larger use case, see MongoDB,
which stores data using a language-neutral binary-encoded
serialization of JSON-like documents, and its
PyMongo interface.
round, math, random, etc.| Literal | Interpretation |
|---|---|
Integers are written as strings of decimal digits. Floating-point numbers have a decimal point and/or an optional signed exponent introduced by aneorEand followed by an optional sign. If you write a number with a decimal point or exponent, Python makes it a floating-point object and uses floating-point (not integer) math when the object is used in an expression. Floating-point numbers are implemented as C “doubles” in standard CPython, and therefore get as much precision as the C compiler used to build the Python interpreter gives to doubles.
In Python 2.X there are two integer types, normal (often 32 bits) and long (unlimited precision), and an integer may end in anlorLto force it to become a long integer. Because integers are automatically converted to long integers when their values overflow their allocated bits, you never need to type the letter L yourself — Python automatically converts up to long integer when extra precision is needed.
In Python 3.X, the normal and long integer types have been merged — there is only integer, which automatically supports the unlimited precision of Python 2.X’s separate long integer type. Because of this, integers can no longer be coded with a trailinglorL, and integers never print with this character either. Apart from this, most programs are unaffected by this change, unless they do type testing that checks for 2.X long integers.
Integers may be coded in decimal (base 10), hexadecimal (base 16), octal (base 8), or binary (base 2), the last three of which are common in some programming domains. Hexadecimals start with a leading0xor0X, followed by a string of hexadecimal digits (0–9andA–F). Hex digits may be coded in lower- or uppercase. Octal literals start with a leading0oor0O(zero and lower- or uppercase letter o), followed by a string of digits (0–7). In 2.X, octal literals can also be coded with just a leading0, but not in 3.X — this original octal form is too easily confused with decimal, and is replaced by the new0oformat, which can also be used in 2.X as of 2.6. Binary literals, new as of 2.6 and 3.0, begin with a leading0bor0B, followed by binary digits (0–1).Note that all of these literals produce integer objects in program code; they are just alternative syntaxes for specifying values. The built-in callshex(I),oct(I), andbin(I)convert an integer to its representation string in these three bases, andint(str,base)converts a runtime string to an integer per a given base.
Python complex literals are written asrealpart+imaginarypart, where theimaginarypartis terminated with ajorJ. Therealpartis technically optional, so theimaginarypartmay appear on its own. Internally, complex numbers are implemented as pairs of floating-point numbers, but all numeric operations perform complex math when applied to complex numbers. Complex numbers may also be created with thecomplex(real,imag)built-in call.
As we’ll see later in this chapter, there are additional numeric types at the end of Table 5-1 that serve more advanced or specialized roles. You create some of these by calling functions in imported modules (e.g., decimals and fractions), and others have literal syntax all their own (e.g., sets).
+,-,*,/,>>,**,&, etc.
pow,abs,round,int,hex,bin, etc.
random,math, etc.
| Operators | Description |
|---|---|
X != Y or X
<> Y. In Python 3.X, the latter of these options is
removed because it is redundant. In either version, best practice is
to use X != Y for all value
inequality tests.`X` works the same as repr(X) and converts objects to display strings. Due to its
obscurity, this expression is removed in Python 3.X; use the more
readable str and
repr built-in functions,
described in “Numeric Display Formats.”X // Y floor division
expression always truncates fractional remainders in both Python 2.X
and 3.X. The X / Y expression
performs true division in 3.X (retaining remainders) and classic
division in 2.X (truncating for integers). See “Division: Classic, Floor, and True”.[...] is used
for both list literals and list comprehension expressions. The
latter of these performs an implied loop and collects expression
results in a new list. See Chapter 4, Chapter 14, and Chapter 20 for examples.(...) is used
for tuples and expression grouping, as well as generator
expressions — a form of list comprehension that produces results on
demand, instead of building a result list. See Chapter 4 and Chapter 20 for examples. The
parentheses may sometimes be omitted in all three contexts. When a
tuple’s parentheses are omitted, the comma
separating its items acts like a lowest-precedence operator if not
otherwise significant.{...} is used
for dictionary literals, and in Python 3.X and 2.7 for set literals
and both dictionary and set comprehensions. See the set coverage in
this chapter as well as Chapter 4, Chapter 8, Chapter 14, and Chapter 20 for examples.yield and ternary
if/else selection expressions are available
in Python 2.5 and later. The former returns send(...) arguments in generators; the
latter is shorthand for a multiline if statement. yield requires parentheses if not alone on
the right side of an assignment statement.X < Y < Z produces the same result
as X < Y and Y < Z. See
“Comparisons: Normal and Chained” for
details.X[I:J:K] is equivalent to indexing with a
slice object: X[slice(I, J,
K)].sorted(aDict.items()) is one
possible replacement.A * B + C * D
(X + Y) * Z X + (Y * Z)
40 + 3.14
>>>40 + 3.14# Integer to float, float math/result43.14
>>>int(3.1415)# Truncates float to integer3 >>>float(3)# Converts integer to float3.0
%python>>>a = 3# Name created: not declared ahead of time>>>b = 4
>>>a + 1, a − 1# Addition (3 + 1), subtraction (3 − 1)(4, 2) >>>b * 3, b / 2# Multiplication (4 * 3), division (4 / 2, 3.X result)(12, 2.0) >>>a % 2, b ** 2# Modulus (remainder), power (4 ** 2)(1, 16) >>>2 + 4.0, 2.0 ** b# Mixed-type conversions(6.0, 16.0)
>>> c * 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'c' is not defined>>>b / 2 + a# Same as ((4 / 2) + 3) [use 2.0 in 2.X]5.0 >>>b / (2.0 + a)# Same as (4 / (2.0 + 3)) [use print before 2.7]0.8
>>>b / (2.0 + a)# Pythons <= 2.6: echoes give more (or fewer) digits0.80000000000000004 >>>print(b / (2.0 + a))# But print rounds off digits0.8
>>> 1 / 2.0
0.5>>>num = 1 / 3.0>>>num# Auto-echoes0.3333333333333333 >>>print(num)# Print explicitly0.3333333333333333 >>>'%e' % num# String formatting expression'3.333333e-01' >>>'%4.2f' % num# Alternative floating-point format'0.33' >>>'{0:4.2f}'.format(num)# String formatting method: Python 2.6, 3.0, and later'0.33'
>>>1 < 2# Less thanTrue >>>2.0 >= 1# Greater than or equal: mixed-type 1 converted to 1.0True >>>2.0 == 2.0# Equal valueTrue >>>2.0 != 2.0# Not equal valueFalse
>>>X = 2>>>Y = 4>>>Z = 6
>>>X < Y < Z# Chained comparisons: range testsTrue >>>X < Y and Y < ZTrue
>>>X < Y > ZFalse >>>X < Y and Y > ZFalse >>>1 < 2 < 3.0 < 4True >>>1 > 2 > 3.0 > 4False
>>>1 == 2 < 3# Same as: 1 == 2 and 2 < 3False# Not same as: False < 3 (which means 0 < 3, which is true!)
>>>1.1 + 2.2 == 3.3# Shouldn't this be True?...False >>>1.1 + 2.2# Close to 3.3, but not exactly: limited precision3.3000000000000003 >>>int(1.1 + 2.2) == int(3.3)# OK if convert: see also round, floor, trunc aheadTrue# Decimals and fractions (ahead) may help here too
X / YClassic and true division. In Python 2.X, this operator performs classic division, truncating results for integers, and keeping remainders (i.e., fractional parts) for floating-point numbers. In Python 3.X, it performs true division, always keeping remainders in floating-point results, regardless of types.
X // Y/ now always performs
true division, returning a float result that
includes any remainder, regardless of operand types. The // performs floor
division, which truncates the remainder and returns an integer for
integer operands or a float if any operand is a float./ does classic
division, performing truncating integer division if both operands
are integers and float division (keeping remainders) otherwise. The
// does
floor division and works as it does in 3.X,
performing truncating division for integers and floor division for
floats.C:\code>C:\Python33\python>>> >>>10 / 4# Differs in 3.X: keeps remainder2.5 >>>10 / 4.0# Same in 3.X: keeps remainder2.5 >>>10 // 4# Same in 3.X: truncates remainder2 >>>10 // 4.0# Same in 3.X: truncates to floor2.0 C:\code>C:\Python27\python>>> >>>10 / 4# This might break on porting to 3.X!2 >>>10 / 4.02.5 >>>10 // 4# Use this in 2.X if truncation needed2 >>>10 // 4.02.0
X = Y // Z# Always truncates, always an int result for ints in 2.X and 3.XX = Y / float(Z)# Guarantees float division with remainder in either 2.X or 3.X
C:\code>C:\Python27\python>>>from __future__ import division# Enable 3.X "/" behavior>>>10 / 42.5 >>>10 // 4# Integer // is the same in both2
>>>import math>>>math.floor(2.5)# Closest number below value2 >>>math.floor(-2.5)-3 >>>math.trunc(2.5)# Truncate fractional part (toward zero)2 >>>math.trunc(-2.5)-2
C:\code>c:\python33\python>>>5 / 2, 5 / −2(2.5, −2.5) >>>5 // 2, 5 // −2# Truncates to floor: rounds to first lower integer(2, −3)# 2.5 becomes 2, −2.5 becomes −3>>>5 / 2.0, 5 / −2.0(2.5, −2.5) >>>5 // 2.0, 5 // −2.0# Ditto for floats, though result is float too(2.0, −3.0)
C:code>c:\python27\python>>>5 / 2, 5 / −2# Differs in 3.X(2, −3) >>>5 // 2, 5 // −2# This and the rest are the same in 2.X and 3.X(2, −3) >>>5 / 2.0, 5 / −2.0(2.5, −2.5) >>>5 // 2.0, 5 // −2.0(2.0, −3.0)
C:\code>c:\python33\python>>>import math>>>5 / −2# Keep remainder−2.5 >>>5 // −2# Floor below result-3 >>>math.trunc(5 / −2)# Truncate instead of floor (same as int())−2 C:\code>c:\python27\python>>>import math>>>5 / float(−2)# Remainder in 2.X−2.5 >>>5 / −2, 5 // −2# Floor in 2.X(−3, −3) >>>math.trunc(5 / float(−2))# Truncate in 2.X−2
>>>(5 / 2), (5 / 2.0), (5 / −2.0), (5 / −2)# 3.X true division(2.5, 2.5, −2.5, −2.5) >>>(5 // 2), (5 // 2.0), (5 // −2.0), (5 // −2)# 3.X floor division(2, 2.0, −3.0, −3) >>>(9 / 3), (9.0 / 3), (9 // 3), (9 // 3.0)# Both(3.0, 3.0, 3, 3.0)
>>>(5 / 2), (5 / 2.0), (5 / −2.0), (5 / −2)# 2.X classic division (differs)(2, 2.5, −2.5, −3) >>>(5 // 2), (5 // 2.0), (5 // −2.0), (5 // −2)# 2.X floor division (same)(2, 2.0, −3.0, −3) >>>(9 / 3), (9.0 / 3), (9 // 3), (9 // 3.0)# Both(3, 3.0, 3, 3.0)
>>>999999999999999999999999999999 + 1# 3.X1000000000000000000000000000000
>>>999999999999999999999999999999 + 1# 2.X1000000000000000000000000000000L
>>>2 ** 2001606938044258990275541962092341162602522202993782792835301376 >>>2 ** 2001606938044258990275541962092341162602522202993782792835301376L
>>>1j * 1J(-1+0j) >>>2 + 1j * 3(2+3j) >>>(2 + 1j) * 3(6+3j)
>>>0o1, 0o20, 0o377# Octal literals: base 8, digits 0-7 (3.X, 2.6+)(1, 16, 255) >>>0x01, 0x10, 0xFF# Hex literals: base 16, digits 0-9/A-F (3.X, 2.X)(1, 16, 255) >>>0b1, 0b10000, 0b11111111# Binary literals: base 2, digits 0-1 (3.X, 2.6+)(1, 16, 255)
>>>0xFF, (15 * (16 ** 1)) + (15 * (16 ** 0))# How hex/binary map to decimal(255, 255) >>>0x2F, (2 * (16 ** 1)) + (15 * (16 ** 0))(47, 47) >>>0xF, 0b1111, (1*(2**3) + 1*(2**2) + 1*(2**1) + 1*(2**0))(15, 15, 15)
>>>oct(64), hex(64), bin(64)# Numbers=>digit strings('0o100', '0x40', '0b1000000')
>>>64, 0o100, 0x40, 0b1000000# Digits=>numbers in scripts and strings(64, 64, 64, 64) >>>int('64'), int('100', 8), int('40', 16), int('1000000', 2)(64, 64, 64, 64) >>>int('0x40', 16), int('0b1000000', 2)# Literal forms supported too(64, 64)
>>> eval('64'), eval('0o100'), eval('0x40'), eval('0b1000000')
(64, 64, 64, 64)>>>'{0:o}, {1:x}, {2:b}'.format(64, 64, 64)# Numbers=>digits, 2.6+'100, 40, 1000000' >>>'%o, %x, %x, %X' % (64, 64, 255, 255)# Similar, in all Pythons'100, 40, ff, FF'
>>>0o1, 0o20, 0o377# New octal format in 2.6+ (same as 3.X)(1, 16, 255) >>>01, 020, 0377# Old octal literals in all 2.X (error in 3.X)(1, 16, 255)
>>>X = 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFF>>>X5192296858534827628530496329220095 >>>oct(X)'0o17777777777777777777777777777777777777' >>>bin(X)'0b111111111111111111111111111111111111111111111111111111111...and so on...11111'
>>>x = 1# 1 decimal is 0001 in bits>>>x << 2# Shift left 2 bits: 01004 >>>x | 2# Bitwise OR (either bit=1): 00113 >>>x & 1# Bitwise AND (both bits=1): 00011
>>>X = 0b0001# Binary literals>>>X << 2# Shift left4 >>>bin(X << 2)# Binary digits string'0b100' >>>bin(X | 0b010)# Bitwise OR: either'0b11' >>>bin(X & 0b1)# Bitwise AND: both'0b1'
>>>X = 0xFF# Hex literals>>>bin(X)'0b11111111' >>>X ^ 0b10101010# Bitwise XOR: either but not both85 >>>bin(X ^ 0b10101010)'0b1010101' >>>int('01010101', 2)# Digits=>number: string to int per base85 >>>hex(85)# Number=>digits: Hex digit string'0x55'
>>>X = 99>>>bin(X), X.bit_length(), len(bin(X)) - 2('0b1100011', 7, 7) >>>bin(256), (256).bit_length(), len(bin(256)) - 2('0b100000000', 9, 9)
>>>import math>>>math.pi, math.e# Common constants(3.141592653589793, 2.718281828459045) >>>math.sin(2 * math.pi / 180)# Sine, tangent, cosine0.03489949670250097 >>>math.sqrt(144), math.sqrt(2)# Square root(12.0, 1.4142135623730951) >>>pow(2, 4), 2 ** 4, 2.0 ** 4.0# Exponentiation (power)(16, 16, 16.0) >>>abs(-42.0), sum((1, 2, 3, 4))# Absolute value, summation(42.0, 10) >>>min(3, 1, 2, 4), max(3, 1, 2, 4)# Minimum, maximum(1, 4)
>>>math.floor(2.567), math.floor(-2.567)# Floor (next-lower integer)(2, −3) >>>math.trunc(2.567), math.trunc(−2.567)# Truncate (drop decimal digits)(2, −2) >>>int(2.567), int(−2.567)# Truncate (integer conversion)(2, −2) >>>round(2.567), round(2.467), round(2.567, 2)# Round (Python 3.X version)(3, 2, 2.57) >>>'%.1f' % 2.567, '{0:.2f}'.format(2.567)# Round for display (Chapter 7)('2.6', '2.57')
>>> (1 / 3.0), round(1 / 3.0, 2), ('%.2f' % (1 / 3.0))
(0.3333333333333333, 0.33, '0.33')>>>import math>>>math.sqrt(144)# Module12.0 >>>144 ** .5# Expression12.0 >>>pow(144, .5)# Built-in12.0 >>>math.sqrt(1234567890)# Larger numbers35136.41828644462 >>>1234567890 ** .535136.41828644462 >>>pow(1234567890, .5)35136.41828644462
>>>import random>>>random.random()0.5566014960423105 >>>random.random()# Random floats, integers, choices, shuffles0.051308506597373515 >>>random.randint(1, 10)5 >>>random.randint(1, 10)9
>>>random.choice(['Life of Brian', 'Holy Grail', 'Meaning of Life'])'Holy Grail' >>>random.choice(['Life of Brian', 'Holy Grail', 'Meaning of Life'])'Life of Brian' >>>suits = ['hearts', 'clubs', 'diamonds', 'spades']>>>random.shuffle(suits)>>>suits['spades', 'hearts', 'diamonds', 'clubs'] >>>random.shuffle(suits)>>>suits['clubs', 'diamonds', 'hearts', 'spades']
>>>0.1 + 0.1 + 0.1 - 0.3# Python 3.35.551115123125783e-17
>>>print(0.1 + 0.1 + 0.1 - 0.3)# Earlier Pythons (3.3. differs)5.55111512313e-17
>>>from decimal import Decimal>>>Decimal('0.1') + Decimal('0.1') + Decimal('0.1') - Decimal('0.3')Decimal('0.0')
>>> Decimal('0.1') + Decimal('0.10') + Decimal('0.10') - Decimal('0.30')
Decimal('0.00')>>> Decimal(0.1) + Decimal(0.1) + Decimal(0.1) - Decimal(0.3)
Decimal('2.775557561565156540423631668E-17')>>>import decimal>>>decimal.Decimal(1) / decimal.Decimal(7)# Default: 28 digitsDecimal('0.1428571428571428571428571429') >>>decimal.getcontext().prec = 4# Fixed precision>>>decimal.Decimal(1) / decimal.Decimal(7)Decimal('0.1429') >>>Decimal(0.1) + Decimal(0.1) + Decimal(0.1) - Decimal(0.3)# Closer to 0Decimal('1.110E-17')
>>>1999 + 1.33# This has more digits in memory than displayed in 3.32000.33 >>> >>>decimal.getcontext().prec = 2>>>pay = decimal.Decimal(str(1999 + 1.33))>>>payDecimal('2000.33')
C:\code>C:\Python33\python>>>import decimal>>>decimal.Decimal('1.00') / decimal.Decimal('3.00')Decimal('0.3333333333333333333333333333') >>> >>>with decimal.localcontext() as ctx:...ctx.prec = 2...decimal.Decimal('1.00') / decimal.Decimal('3.00')... Decimal('0.33') >>> >>>decimal.Decimal('1.00') / decimal.Decimal('3.00')Decimal('0.3333333333333333333333333333')
>>>from fractions import Fraction>>>x = Fraction(1, 3)# Numerator, denominator>>>y = Fraction(4, 6)# Simplified to 2, 3 by gcd>>>xFraction(1, 3) >>>yFraction(2, 3) >>>print(y)2/3
>>>x + yFraction(1, 1) >>>x − y# Results are exact: numerator, denominatorFraction(−1, 3) >>>x * yFraction(2, 9)
>>>Fraction('.25')Fraction(1, 4) >>>Fraction('1.25')Fraction(5, 4) >>> >>>Fraction('.25') + Fraction('1.25')Fraction(3, 2)
>>>a = 1 / 3.0# Only as accurate as floating-point hardware>>>b = 4 / 6.0# Can lose precision over many calculations>>>a0.3333333333333333 >>>b0.6666666666666666 >>>a + b1.0 >>>a - b-0.3333333333333333 >>>a * b0.2222222222222222
>>>0.1 + 0.1 + 0.1 - 0.3# This should be zero (close, but not exact)5.551115123125783e-17 >>>from fractions import Fraction>>>Fraction(1, 10) + Fraction(1, 10) + Fraction(1, 10) - Fraction(3, 10)Fraction(0, 1) >>>from decimal import Decimal>>>Decimal('0.1') + Decimal('0.1') + Decimal('0.1') - Decimal('0.3')Decimal('0.0')
>>>1 / 3# Use a ".0" in Python 2.X for true "/"0.3333333333333333 >>>Fraction(1, 3)# Numeric accuracy, two waysFraction(1, 3) >>>import decimal>>>decimal.getcontext().prec = 2>>>Decimal(1) / Decimal(3)Decimal('0.33')
>>>(1 / 3) + (6 / 12)# Use a ".0" in Python 2.X for true "/"0.8333333333333333 >>>Fraction(6, 12)# Automatically simplifiedFraction(1, 2) >>>Fraction(1, 3) + Fraction(6, 12)Fraction(5, 6) >>>decimal.Decimal(str(1/3)) + decimal.Decimal(str(6/12))Decimal('0.83') >>>1000.0 / 12345678908.100000073710001e-07 >>>Fraction(1000, 1234567890)# Substantially simpler!Fraction(100, 123456789)
>>>(2.5).as_integer_ratio()# float object method(5, 2) >>>f = 2.5>>>z = Fraction(*f.as_integer_ratio())# Convert float -> fraction: two args>>>z# Same as Fraction(5, 2)Fraction(5, 2) >>>x# x from prior interactionFraction(1, 3) >>>x + zFraction(17, 6)# 5/2 + 1/3 = 15/6 + 2/6>>>float(x)# Convert fraction -> float0.3333333333333333 >>>float(z)2.5 >>>float(x + z)2.8333333333333335 >>>17 / 62.8333333333333335 >>>Fraction.from_float(1.75)# Convert float -> fraction: other wayFraction(7, 4) >>>Fraction(*(1.75).as_integer_ratio())Fraction(7, 4)
>>>xFraction(1, 3) >>>x + 2# Fraction + int -> FractionFraction(7, 3) >>>x + 2.0# Fraction + float -> float2.3333333333333335 >>>x + (1./3)# Fraction + float -> float0.6666666666666666 >>>x + (4./3)1.6666666666666665 >>>x + Fraction(4, 3)# Fraction + Fraction -> FractionFraction(5, 3)
>>>4.0 / 31.3333333333333333 >>>(4.0 / 3).as_integer_ratio()# Precision loss from float(6004799503160661, 4503599627370496) >>>xFraction(1, 3) >>>a = x + Fraction(*(4.0 / 3).as_integer_ratio())>>>aFraction(22517998136852479, 13510798882111488) >>>22517998136852479 / 13510798882111488.# 5 / 3 (or close to it!)1.6666666666666667 >>>a.limit_denominator(10)# Simplify to closest fractionFraction(5, 3)
>>>x = set('abcde')>>>y = set('bdxyz')
>>>xset(['a', 'c', 'b', 'e', 'd'])# Pythons <= 2.6 display format
>>>x − y# Differenceset(['a', 'c', 'e']) >>>x | y# Unionset(['a', 'c', 'b', 'e', 'd', 'y', 'x', 'z']) >>>x & y# Intersectionset(['b', 'd']) >>>x ^ y# Symmetric difference (XOR)set(['a', 'c', 'e', 'y', 'x', 'z']) >>>x > y, x < y# Superset, subset(False, False)
>>>'e' in x# Membership (sets)True >>>'e' in 'Camelot', 22 in [11, 22, 33]# But works on other types too(True, True)
>>>z = x.intersection(y)# Same as x & y>>>zset(['b', 'd']) >>>z.add('SPAM')# Insert one item>>>zset(['b', 'd', 'SPAM']) >>>z.update(set(['X', 'Y']))# Merge: in-place union>>>zset(['Y', 'X', 'b', 'd', 'SPAM']) >>>z.remove('b')# Delete one item>>>zset(['Y', 'X', 'd', 'SPAM'])
>>> for item in set('abc'): print(item * 3)
aaa
ccc
bbb>>>S = set([1, 2, 3])>>>S | set([3, 4])# Expressions require both to be setsset([1, 2, 3, 4]) >>>S | [3, 4]TypeError: unsupported operand type(s) for |: 'set' and 'list' >>>S.union([3, 4])# But their methods allow any iterableset([1, 2, 3, 4]) >>>S.intersection((1, 3, 5))set([1, 3]) >>>S.issubset(range(-5, 5))True
set([1, 2, 3, 4])# Built-in call (all){1, 2, 3, 4}# Newer set literals (2.7, 3.X)
C:\code>c:\python33\python>>>set([1, 2, 3, 4])# Built-in: same as in 2.6{1, 2, 3, 4} >>>set('spam')# Add all items in an iterable{'s', 'a', 'p', 'm'} >>>{1, 2, 3, 4}# Set literals: new in 3.X (and 2.7){1, 2, 3, 4} >>>S = {'s', 'p', 'a', 'm'}>>>S{'s', 'a', 'p', 'm'} >>>S.add('alot')# Methods work as before>>>S{'s', 'a', 'p', 'alot', 'm'}
>>>S1 = {1, 2, 3, 4}>>>S1 & {1, 3}# Intersection{1, 3} >>>{1, 5, 3, 6} | S1# Union{1, 2, 3, 4, 5, 6} >>>S1 - {1, 3, 4}# Difference{2} >>>S1 > {1, 3}# SupersetTrue
>>>S1 - {1, 2, 3, 4}# Empty sets print differentlyset() >>>type({})# Because {} is an empty dictionary<class 'dict'> >>>S = set()# Initialize an empty set>>>S.add(1.23)>>>S{1.23}
>>>{1, 2, 3} | {3, 4}{1, 2, 3, 4} >>>{1, 2, 3} | [3, 4]TypeError: unsupported operand type(s) for |: 'set' and 'list' >>>{1, 2, 3}.union([3, 4]){1, 2, 3, 4} >>>{1, 2, 3}.union({3, 4}){1, 2, 3, 4} >>>{1, 2, 3}.union(set([3, 4])){1, 2, 3, 4} >>>{1, 2, 3}.intersection((1, 3, 5)){1, 3} >>>{1, 2, 3}.issubset(range(-5, 5))True
>>>S{1.23} >>>S.add([1, 2, 3])# Only immutable objects work in a setTypeError: unhashable type: 'list' >>>S.add({'a':1})TypeError: unhashable type: 'dict' >>>S.add((1, 2, 3))>>>S# No list or dict, but tuple OK{1.23, (1, 2, 3)} >>>S | {(4, 5, 6), (1, 2, 3)}# Union: same as S.union(...){1.23, (4, 5, 6), (1, 2, 3)} >>>(1, 2, 3) in S# Membership: by complete valuesTrue >>>(1, 4, 3) in SFalse
>>>{x ** 2 for x in [1, 2, 3, 4]}# 3.X/2.7 set comprehension{16, 1, 4, 9}
>>>{x for x in 'spam'}# Same as: set('spam'){'m', 's', 'p', 'a'} >>>{c * 4 for c in 'spam'}# Set of collected expression results{'pppp', 'aaaa', 'ssss', 'mmmm'} >>>{c * 4 for c in 'spamham'}{'pppp', 'aaaa', 'hhhh', 'ssss', 'mmmm'} >>>S = {c * 4 for c in 'spam'}>>>S | {'mmmm', 'xxxx'}{'pppp', 'xxxx', 'mmmm', 'aaaa', 'ssss'} >>>S & {'mmmm', 'xxxx'}{'mmmm'}
>>>L = [1, 2, 1, 3, 2, 4, 5]>>>set(L){1, 2, 3, 4, 5} >>>L = list(set(L))# Remove duplicates>>>L[1, 2, 3, 4, 5] >>>list(set(['yy', 'cc', 'aa', 'xx', 'dd', 'aa']))# But order may change['cc', 'xx', 'yy', 'dd', 'aa']
>>>set([1, 3, 5, 7]) - set([1, 2, 4, 5, 6])# Find list differences{3, 7}>>>set('abcdefg') - set('abdghij')# Find string differences{'c', 'e', 'f'} >>>set('spam') - set(['h', 'a', 'm'])# Find differences, mixed{'p', 's'} >>>set(dir(bytes)) - set(dir(bytearray))# In bytes but not bytearray{'__getnewargs__'} >>>set(dir(bytearray)) - set(dir(bytes)){'append', 'copy', '__alloc__', '__imul__', 'remove', 'pop', 'insert',...more...]
>>>L1, L2 = [1, 3, 5, 2, 4], [2, 5, 3, 4, 1]>>>L1 == L2# Order matters in sequencesFalse >>>set(L1) == set(L2)# Order-neutral equalityTrue >>>sorted(L1) == sorted(L2)# Similar but results orderedTrue >>>'spam' == 'asmp', set('spam') == set('asmp'), sorted('spam') == sorted('asmp')(False, True, True)
>>>engineers = {'bob', 'sue', 'ann', 'vic'}>>>managers = {'tom', 'sue'}>>>'bob' in engineers# Is bob an engineer?True >>>engineers & managers# Who is both engineer and manager?{'sue'} >>>engineers | managers# All people in either category{'bob', 'tom', 'sue', 'vic', 'ann'} >>>engineers - managers# Engineers who are not managers{'vic', 'ann', 'bob'} >>>managers - engineers# Managers who are not engineers{'tom'} >>>engineers > managers# Are all managers engineers? (superset)False >>>{'bob', 'sue'} < engineers# Are both engineers? (subset)True >>>(managers | engineers) > managers# All people is a superset of managersTrue >>>managers ^ engineers# Who is in one but not both?{'tom', 'vic', 'ann', 'bob'} >>>(managers | engineers) - (managers ^ engineers)# Intersection!{'sue'}
>>>type(True)<class 'bool'> >>>isinstance(True, int)True >>>True == 1# Same valueTrue >>>True is 1# But a different object: see the next chapterFalse >>>True or False# Same as: 1 or 0True >>>True + 4# (Hmmm)5
2 * (3
+ 4) in Python?2 *
3 + 4 in Python?2 +
3 * 4 in Python?1 + 2.0 + 3?14, the
result of 2 * 7, because the parentheses force the addition to happen
before the multiplication.10, the
result of 6 + 4. Python’s operator precedence rules are applied in the
absence of parentheses, and multiplication has higher precedence than
(i.e., happens before) addition, per Table 5-2.14,
the result of 2 + 12, for the same precedence reasons as in the prior
question.math module. To find a
number’s square root, import math
and call math.sqrt(N). To get a
number’s square, use either the exponent expression X ** 2 or the built-in function pow(X, 2). Either of these last two can also
compute the square root when given a power of 0.5 (e.g., X **
.5).int(N) and math.trunc(N) functions truncate, and the round(N, digits) function rounds. We can also compute the
floor with math.floor(N) and round for display with string
formatting operations.float(I) function converts an integer to a floating
point; mixing an integer with a floating point within an expression
will result in a conversion as well. In some sense, Python 3.X
/ division converts too — it always
returns a floating-point result that includes the remainder, even if
both operands are integers.oct(I) and hex(I) built-in functions return the octal and
hexadecimal string forms for an integer. The bin(I) call also returns a number’s binary digits
string in Pythons 2.6, 3.0, and later. The % string formatting expression and format string method also provide targets
for some such conversions.int(S, base) function can be used to convert from octal
and hexadecimal strings to normal integers (pass in 8, 16, or
2 for the base). The eval(S) function can be used for this purpose too,
but it’s more expensive to run and can have security issues. Note that
integers are always stored in binary form in computer memory; these
are just display string format conversions.A variable (also known in Python as a name), likea, is created when your code first assigns it a value. Future assignments change the value of the already created name. Technically, Python detects some names before your code runs, but you can think of it as though initial assignments make variables.
A variable never has any type information or constraints associated with it. The notion of type lives with objects, not names. Variables are generic in nature; they always simply refer to a particular object at a particular point in time.
>>>a = 3# Assign a name to an object
3.a, if
it does not yet exist.a to the
new object 3.>>>a = 3# It's an integer>>>a = 'spam'# Now it's a string>>>a = 1.23# Now it's a floating point
>>>a = 3>>>a = 'spam'
>>>x = 42>>>x = 'shrubbery'# Reclaim 42 now (unless referenced elsewhere) >>>x = 3.1415# Reclaim 'shrubbery' now >>>x = [1, 2, 3]# Reclaim 3.1415 now
A still prints as
"spam". When B is assigned to the string "shrubbery", all that happens is that the
variable B is reset to point to the
new string object. A and B initially share (i.e., reference/point to)
the same single string object "spam", but two names are never linked
together in Python. Thus, setting B
to a different object has no effect on A. The same would be true if the last
statement here were B = B +
'shrubbery', by the way — the concatenation would make a new
object for its result, which would then be assigned to B only. We can never overwrite a string (or
number, or tuple) in place, because strings are immutable.A now prints as ["shrubbery"]. Technically, we haven’t
really changed either A or B; instead, we’ve changed part of the object
they both reference (point to) by overwriting that object in place
through the variable B. Because
A references the same object as
B, the update is reflected in
A as well.A still prints as
["spam"]. The in-place assignment
through B has no effect this time
because the slice expression made a copy of the list object before it
was assigned to B. After the second
assignment statement, there are two different list objects that have
the same value (in Python, we say they are ==, but not is). The third statement changes the value
of the list object pointed to by B,
but not that pointed to by A.1 Readers with a background in C may find Python references similar to C pointers (memory addresses). In fact, references are implemented as pointers, and they often serve the same roles, especially with objects that can be changed in place (more on this later). However, because references are always automatically dereferenced when used, you can never actually do anything useful with a reference itself; this is a feature that eliminates a vast category of C bugs. But you can think of Python references as C “void*” pointers, which are automatically followed whenever used.
str is used for Unicode text (including
ASCII), bytes is used for binary
data (including encoded text), and bytearray is a mutable variant of bytes. Files
work in two modes: text, which represents content as str and
implements Unicode encodings, and binary, which
deals in raw bytes and does no
data translation.unicode strings represent Unicode text, str strings handle both 8-bit text and
binary data, and bytearray is
available in 2.6 and later as a back-port from 3.X. Normal files’
content is simply bytes represented as str, but a codecs module opens Unicode text files,
handles encodings, and represents content as unicode objects.| Operation | Interpretation |
|---|---|
'spa"m'"spa'm"'''... spam
...''', """... spam
...""""s\tp\na\0m"r"C:\new\test.spm"b'sp\x01am'u'eggs\u0020spam'>>> 'shrubbery', "shrubbery"
('shrubbery', 'shrubbery')>>> 'knight"s', "knight's"
('knight"s', "knight's")>>>title = "Meaning " 'of' " Life"# Implicit concatenation>>>title'Meaning of Life'
>>> 'knight\'s', "knight\"s"
("knight's", 'knight"s')>>> s = 'a\nb\tc'>>>s'a\nb\tc' >>>print(s)a b c
>>> len(s)
5| Escape | Meaning |
|---|---|
a The | |
>>>s = 'a\0b\0c'>>>s'a\x00b\x00c' >>>len(s)5
>>>s = '\001\002\x03'>>>s'\x01\x02\x03' >>>len(s)3
>>>S = "s\tp\na\x00m">>>S's\tp\na\x00m' >>>len(S)7 >>>print(S)s p a m
>>>x = "C:\py\code"# Keeps \ literally (and displays it as \\)>>>x'C:\\py\\code' >>>len(x)10
myfile = open('C:\new\text.dat', 'w')myfile = open(r'C:\new\text.dat', 'w')
myfile = open('C:\\new\\text.dat', 'w')>>>path = r'C:\new\text.dat'>>>path# Show as Python code'C:\\new\\text.dat' >>>print(path)# User-friendly formatC:\new\text.dat >>>len(path)# String length15
>>>mantra = """Always look...on the bright...side of life.""">>> >>>mantra'Always look\n on the bright\nside of life.'
>>> print(mantra)
Always look
on the bright
side of life.>>>menu = """spam # comments here added to string!...eggs # ditto...""">>>menu'spam # comments here added to string!\neggs # ditto\n' >>>menu = (..."spam\n" # comments here ignored..."eggs\n" # but newlines not automatic...)>>>menu'spam\neggs\n'
X = 1
"""
import os # Disable this code temporarily
print(os.getcwd())
"""
Y = 2%python>>>len('abc')# Length: number of items3 >>>'abc' + 'def'# Concatenation: a new string'abcdef' >>>'Ni!' * 4# Repetition: like "Ni!" + "Ni!" + ...'Ni!Ni!Ni!Ni!'
>>>print('-------...more...---')# 80 dashes, the hard way>>>print('-' * 80)# 80 dashes, the easy way
>>>myjob = "hacker">>>for c in myjob: print(c, end=' ')# Step through items, print each (3.X form)... h a c k e r >>>"k" in myjob# FoundTrue >>>"z" in myjob# Not foundFalse >>>'spam' in 'abcspamdef'# Substring search, no position returnedTrue
>>>S = 'spam'>>>S[0], S[−2]# Indexing from front or end('s', 'a') >>>S[1:3], S[1:], S[:−1]# Slicing: extract a section('pa', 'pam', 'spa')
S[i]) fetches components at offsets:
- The first item is at offset 0.
- Negative indexes mean to count backward from the end or right.
S[0]fetches the first item.S[−2]fetches the second item from the end (likeS[len(S)−2]).
S[i:j]) extracts contiguous sections of
sequences:
- The upper bound is noninclusive.
- Slice boundaries default to 0 and the sequence length, if omitted.
S[1:3]fetches items at offsets 1 up to but not including 3.S[1:]fetches items at offset 1 through the end (the sequence length).S[:3]fetches items at offset 0 up to but not including 3.S[:−1]fetches items at offset 0 up to but not including the last item.S[:]fetches items at offsets 0 through the end — making a top-level copy ofS.
S[i:j:k]) accepts a step (or stride) k, which defaults to +1:
- Allows for skipping items and reversing order — see the next section.
>>>S = 'abcdefghijklmnop'>>>S[1:10:2]# Skipping items'bdfhj' >>>S[::2]'acegikmo'
>>>S = 'hello'>>>S[::−1]# Reversing items'olleh'
>>>S = 'abcedfg'>>>S[5:1:−1]# Bounds roles differ'fdec'
>>>'spam'[1:3]# Slicing syntax'pa' >>>'spam'[slice(1, 3)]# Slice objects with index syntax + object'pa' >>>'spam'[::-1]'maps' >>>'spam'[slice(None, None, −1)]'maps'
# Python 3.X>>>"42" + 1TypeError: Can't convert 'int' object to str implicitly# Python 2.X>>>"42" + 1TypeError: cannot concatenate 'str' and 'int' objects
>>>int("42"), str(42)# Convert from/to string(42, '42') >>>repr(42)# Convert to as-code string'42'
>>>print(str('spam'), repr('spam'))# 2.X: print str('spam'), repr('spam')spam 'spam' >>>str('spam'), repr('spam')# Raw interactive echo displays('spam', "'spam'")
>>>S = "42">>>I = 1>>>S + ITypeError: Can't convert 'int' object to str implicitly >>>int(S) + I# Force addition43 >>>S + str(I)# Force concatenation'421'
>>>str(3.1415), float("1.5")('3.1415', 1.5) >>>text = "1.234E-10">>>float(text)# Shows more digits before 2.7 and 3.11.234e-10
>>>ord('s')115 >>>chr(115)'s'
>>>S = '5'>>>S = chr(ord(S) + 1)>>>S'6' >>>S = chr(ord(S) + 1)>>>S'7'
>>>int('5')5 >>>ord('5') - ord('0')5
>>>B = '1101'# Convert binary digits to integer with ord>>>I = 0>>>while B != '':...I = I * 2 + (ord(B[0]) - ord('0'))...B = B[1:]... >>>I13
>>>int('1101', 2)# Convert binary to integer: built-in13 >>>bin(13)# Convert integer to binary: built-in'0b1101'
>>>S = 'spam'>>>S[0] = 'x'# Raises an error!TypeError: 'str' object does not support item assignment
>>>S = S + 'SPAM!'# To change a string, make a new one>>>S'spamSPAM!' >>>S = S[:4] + 'Burger' + S[−1]>>>S'spamBurger!'
>>>S = 'splot'>>>S = S.replace('pl', 'pamal')>>>S'spamalot'
>>>'That is %d %s bird!' % (1, 'dead')# Format expression: all PythonsThat is 1 dead bird! >>>'That is {0} {1} bird!'.format(1, 'dead')# Format method in 2.6, 2.7, 3.X'That is 1 dead bird!'
object.method(arguments)
Callmethodto processobjectwitharguments.
>>>S = 'spam'>>>result = S.find('pa')# Call the find method to look for 'pa' in string S
>>>S = 'spammy'>>>S = S[:3] + 'xx' + S[5:]# Slice sections from S>>>S'spaxxy'
>>>S = 'spammy'>>>S = S.replace('mm', 'xx')# Replace all mm with xx in S>>>S'spaxxy'
>>> 'aa$bb$cc$dd'.replace('$', 'SPAM')
'aaSPAMbbSPAMccSPAMdd'>>>S = 'xxxxSPAMxxxxSPAMxxxx'>>>where = S.find('SPAM')# Search for position>>>where# Occurs at offset 44 >>>S = S[:where] + 'EGGS' + S[(where+4):]>>>S'xxxxEGGSxxxxSPAMxxxx'
>>>S = 'xxxxSPAMxxxxSPAMxxxx'>>>S.replace('SPAM', 'EGGS')# Replace all'xxxxEGGSxxxxEGGSxxxx' >>>S.replace('SPAM', 'EGGS', 1)# Replace one'xxxxEGGSxxxxSPAMxxxx'
>>>S = 'spammy'>>>L = list(S)>>>L['s', 'p', 'a', 'm', 'm', 'y']
>>>L[3] = 'x'# Works for lists, not strings>>>L[4] = 'x'>>>L['s', 'p', 'a', 'x', 'x', 'y']
>>>S = ''.join(L)>>>S'spaxxy'
>>> 'SPAM'.join(['eggs', 'sausage', 'ham', 'toast'])
'eggsSPAMsausageSPAMhamSPAMtoast'>>>line = 'aaa bbb ccc'>>>col1 = line[0:3]>>>col3 = line[8:]>>>col1'aaa' >>>col3'ccc'
>>>line = 'aaa bbb ccc'>>>cols = line.split()>>>cols['aaa', 'bbb', 'ccc']
>>>line = 'bob,hacker,40'>>>line.split(',')['bob', 'hacker', '40']
>>>line = "i'mSPAMaSPAMlumberjack">>>line.split("SPAM")["i'm", 'a', 'lumberjack']
>>>line = "The knights who say Ni!\n">>>line.rstrip()'The knights who say Ni!' >>>line.upper()'THE KNIGHTS WHO SAY NI!\n' >>>line.isalpha()False >>>line.endswith('Ni!\n')True >>>line.startswith('The')True
>>>line'The knights who say Ni!\n' >>>line.find('Ni') != −1# Search via method call or expressionTrue >>>'Ni' in lineTrue >>>sub = 'Ni!\n'>>>line.endswith(sub)# End test via method call or sliceTrue >>>line[-len(sub):] == subTrue
X.method(arguments)
string.method(X,arguments)
>>>S = 'a+b+c+'>>>x = S.replace('+', 'spam')>>>x'aspambspamcspam'
>>>import string>>>y = string.replace(S, '+', 'spam')>>>y'aspambspamcspam'
'...%s...' % (values)The original technique available since Python’s inception, this form is based upon the C language’s “printf” model, and sees widespread use in much existing code.
'...{}...'.format(values)A newer technique added in Python 2.6 and 3.0, this form is derived in part from a same-named tool in C#/.NET, and overlaps with string formatting expression functionality.
There should be one — and preferably only one — obvious way to do it.
% operator, provide a format string
containing one or more embedded conversion targets, each of which
starts with a % (e.g., %d).% operator, provide the object (or
objects, embedded in a tuple) that you want Python to insert into
the format string on the left in place of the conversion target (or
targets).>>>'That is %d %s bird!' % (1, 'dead')# Format expressionThat is 1 dead bird!
>>>exclamation = 'Ni'>>>'The knights who say %s!' % exclamation# String substitution'The knights who say Ni!' >>>'%d %s %g you' % (1, 'spam', 4.0)# Type-specific substitutions'1 spam 4 you' >>>'%s -- %s -- %s' % (42, 3.14159, [1, 2, 3])# All types match a %s target'42 -- 3.14159 -- [1, 2, 3]'
| Code | Meaning |
|---|---|
%[(keyname)][flags][width][.precision]typecode
−), numeric sign
(+), a blank before positive
numbers and a – for negatives (a
space), and zero fills (0)>>>x = 1234>>>res = 'integers: ...%d...%−6d...%06d' % (x, x, x)>>>res'integers: ...1234...1234 ...001234'
>>>x = 1.23456789>>>x# Shows more digits before 2.7 and 3.11.23456789 >>>'%e | %f | %g' % (x, x, x)'1.234568e+00 | 1.234568 | 1.23457' >>>'%E' % x'1.234568E+00'
>>>'%−6.2f | %05.2f | %+06.1f' % (x, x, x)'1.23 | 01.23 | +001.2' >>>'%s' % x, str(x)('1.23456789', '1.23456789')
>>> '%f, %.2f, %.*f' % (1/3.0, 1/3.0, 4, 1/3.0)
'0.333333, 0.33, 0.3333'>>> '%(qty)d more %(food)s' % {'qty': 1, 'food': 'spam'}
'1 more spam'>>># Template with substitution targets>>>reply = """Greetings...Hello %(name)s!Your age is %(age)s""">>>values = {'name': 'Bob', 'age': 40}# Build up values to substitute>>>print(reply % values)# Perform substitutionsGreetings... Hello Bob! Your age is 40
>>>food = 'spam'>>>qty = 10>>>vars(){'food': 'spam', 'qty': 10,...plus built-in names set by Python...}
>>>'%(qty)d more %(food)s' % vars()# Variables are keys in vars()'10 more spam'
>>>template = '{0}, {1} and {2}'# By position>>>template.format('spam', 'ham', 'eggs')'spam, ham and eggs' >>>template = '{motto}, {pork} and {food}'# By keyword>>>template.format(motto='spam', pork='ham', food='eggs')'spam, ham and eggs' >>>template = '{motto}, {0} and {food}'# By both>>>template.format('ham', motto='spam', food='eggs')'spam, ham and eggs' >>>template = '{}, {} and {}'# By relative position>>>template.format('spam', 'ham', 'eggs')# New in 3.1 and 2.7'spam, ham and eggs'
>>>template = '%s, %s and %s'# Same via expression>>>template % ('spam', 'ham', 'eggs')'spam, ham and eggs' >>>template = '%(motto)s, %(pork)s and %(food)s'>>>template % dict(motto='spam', pork='ham', food='eggs')'spam, ham and eggs'
>>> '{motto}, {0} and {food}'.format(42, motto=3.14, food=[1, 2])
'3.14, 42 and [1, 2]'>>>X = '{motto}, {0} and {food}'.format(42, motto=3.14, food=[1, 2])>>>X'3.14, 42 and [1, 2]' >>>X.split(' and ')['3.14, 42', '[1, 2]'] >>>Y = X.replace('and', 'but under no circumstances')>>>Y'3.14, 42 but under no circumstances [1, 2]'
>>>import sys>>>'My {1[kind]} runs {0.platform}'.format(sys, {'kind': 'laptop'})'My laptop runs win32' >>>'My {map[kind]} runs {sys.platform}'.format(sys=sys,map={'kind': 'laptop'})'My laptop runs win32'
>>>somelist = list('SPAM')>>>somelist['S', 'P', 'A', 'M'] >>>'first={0[0]}, third={0[2]}'.format(somelist)'first=S, third=A' >>>'first={0}, last={1}'.format(somelist[0], somelist[-1])# [-1] fails in fmt 'first=S, last=M' >>>parts = somelist[0], somelist[-1], somelist[1:3]# [1:3] fails in fmt >>>'first={0}, last={1}, middle={2}'.format(*parts)# Or '{}' in 2.7/3.1+"first=S, last=M, middle=['P', 'A']"
{fieldname component !conversionflag :formatspec}fieldname is an optional number or
keyword identifying an argument, which may be omitted to use
relative argument numbering in 2.7, 3.1, and later.component is a string of zero or
more “.name” or
“[index]” references used to fetch
attributes and indexed values of the argument, which may be omitted
to use the whole argument value.conversionflag starts with a
! if present, which is followed
by r, s, or a
to call repr, str, or ascii built-in functions on the value, respectively.formatspec starts with a : if present, which is followed by text
that specifies how the value should be presented, including details
such as field width, alignment, padding, decimal precision, and so
on, and ends with an optional data type code.[[fill]align][sign][#][0][width][,][.precision][typecode]
>>>'{0:10} = {1:10}'.format('spam', 123.4567)# In Python 3.3'spam = 123.4567' >>>'{0:>10} = {1:<10}'.format('spam', 123.4567)' spam = 123.4567 ' >>>'{0.platform:>10} = {1[kind]:<10}'.format(sys, dict(kind='laptop'))' win32 = laptop '
>>>'{:10} = {:10}'.format('spam', 123.4567)'spam = 123.4567' >>>'{:>10} = {:<10}'.format('spam', 123.4567)' spam = 123.4567 ' >>>'{.platform:>10} = {[kind]:<10}'.format(sys, dict(kind='laptop'))' win32 = laptop '
>>>'{0:e}, {1:.3e}, {2:g}'.format(3.14159, 3.14159, 3.14159)'3.141590e+00, 3.142e+00, 3.14159' >>>'{0:f}, {1:.2f}, {2:06.2f}'.format(3.14159, 3.14159, 3.14159)'3.141590, 3.14, 003.14'
>>>'{0:X}, {1:o}, {2:b}'.format(255, 255, 255)# Hex, octal, binary'FF, 377, 11111111' >>>bin(255), int('11111111', 2), 0b11111111# Other to/from binary('0b11111111', 255, 255) >>>hex(255), int('FF', 16), 0xFF# Other to/from hex('0xff', 255, 255) >>>oct(255), int('377', 8), 0o377# Other to/from octal, in 3.X('0o377', 255, 255)# 2.X prints and accepts 0377
>>>'{0:.2f}'.format(1 / 3.0)# Parameters hardcoded'0.33' >>>'%.2f' % (1 / 3.0)# Ditto for expression'0.33' >>>'{0:.{1}f}'.format(1 / 3.0, 4)# Take value from arguments'0.3333' >>>'%.*f' % (4, 1 / 3.0)# Ditto for expression'0.3333'
>>>'{0:.2f}'.format(1.2345)# String method'1.23' >>>format(1.2345, '.2f')# Built-in function'1.23' >>>'%.2f' % 1.2345# Expression'1.23'
print('%s=%s' % ('spam', 42)) # Format expression: in all 2.X/3.X
print('{0}={1}'.format('spam', 42)) # Format method: in 3.0+ and 2.6+
print('{}={}'.format('spam', 42)) # With autonumbering: in 3.1+ and 2.7>>>'%s, %s and %s' % (3.14, 42, [1, 2])# Arbitrary types'3.14, 42 and [1, 2]' >>>'My %(kind)s runs %(platform)s' % {'kind': 'laptop', 'platform': sys.platform}'My laptop runs win32' >>>'My %(kind)s runs %(platform)s' % dict(kind='laptop', platform=sys.platform)'My laptop runs win32' >>>somelist = list('SPAM')>>>parts = somelist[0], somelist[-1], somelist[1:3]>>>'first=%s, last=%s, middle=%s' % parts"first=S, last=M, middle=['P', 'A']"
# Adding specific formatting>>>'%-10s = %10s' % ('spam', 123.4567)'spam = 123.4567' >>>'%10s = %-10s' % ('spam', 123.4567)' spam = 123.4567 ' >>>'%(plat)10s = %(kind)-10s' % dict(plat=sys.platform, kind='laptop')' win32 = laptop '# Floating-point numbers>>>'%e, %.3e, %g' % (3.14159, 3.14159, 3.14159)'3.141590e+00, 3.142e+00, 3.14159' >>>'%f, %.2f, %06.2f' % (3.14159, 3.14159, 3.14159)'3.141590, 3.14, 003.14'# Hex and octal, but not binary (see ahead)>>>'%x, %o' % (255, 255)'ff, 377'
# Hardcoded references in both>>>import sys>>>'My {1[kind]:<8} runs {0.platform:>8}'.format(sys, {'kind': 'laptop'})'My laptop runs win32' >>>'My %(kind)-8s runs %(plat)8s' % dict(kind='laptop', plat=sys.platform)'My laptop runs win32'
# Building data ahead of time in both>>>data = dict(platform=sys.platform, kind='laptop')>>>'My {kind:<8} runs {platform:>8}'.format(**data)'My laptop runs win32' >>>'My %(kind)-8s runs %(platform)8s' % data'My laptop runs win32'
>>>'{0:d}'.format(999999999999)'999999999999' >>>'{0:,d}'.format(999999999999)'999,999,999,999'
>>>'{:,d}'.format(999999999999)'999,999,999,999' >>>'{:,d} {:,d}'.format(9999999, 8888888)'9,999,999 8,888,888' >>>'{:,.2f}'.format(296999.2567)'296,999.26'
>>>from formats import commas, money>>>'%s' % commas(999999999999)'999,999,999,999' >>>'%s %s' % (commas(9999999), commas(8888888))'9,999,999 8,888,888' >>>'%s' % money(296999.2567)'$296,999.26'
>>>[commas(x) for x in (9999999, 8888888)]['9,999,999', '8,888,888'] >>>'%s %s' % tuple(commas(x) for x in (9999999, 8888888))'9,999,999 8,888,888' >>>''.join(commas(x) for x in (9999999, 8888888))'9,999,9998,888,888'
% expression itself (though % can use alternatives)% often has
equivalents)>>>'{0:b}'.format((2 ** 16) − 1)# Expression (only) binary format code'1111111111111111' >>>'%b' % ((2 ** 16) − 1)ValueError: unsupported format character 'b'... >>>bin((2 ** 16) − 1)# But other more general options work too'0b1111111111111111' >>>'%s' % bin((2 ** 16) - 1)# Usable with both method and % expression'0b1111111111111111' >>>'{}'.format(bin((2 ** 16) - 1))# With 2.7/3.1+ relative numbering'0b1111111111111111' >>>'%s' % bin((2 ** 16) - 1)[2:]# Slice off 0b to get exact equivalent'1111111111111111'
>>>'{:,d}'.format(999999999999)# New str.format method feature in 3.1/2.7'999,999,999,999' >>>'%s' % commas(999999999999)# But % is same with simple 8-line function'999,999,999,999'
>>>'{name} {job} {name}'.format(name='Bob', job='dev')'Bob dev Bob' >>>'%(name)s %(job)s %(name)s' % dict(name='Bob', job='dev')'Bob dev Bob'
>>>D = dict(name='Bob', job='dev')>>>'{0[name]} {0[job]} {0[name]}'.format(D)# Method, key references'Bob dev Bob' >>>'{name} {job} {name}'.format(**D)# Method, dict-to-args'Bob dev Bob' >>>'%(name)s %(job)s %(name)s' % D# Expression, key references'Bob dev Bob'
'\n%s<Class %s, address %s:\n%s%s%s>\n' % (...)# Expression'\n{0}<Class {1}, address {2}:\n{3}{4}{5}>\n'.format(...)# Method
>>>'The {0} side {1} {2}'.format('bright', 'of', 'life')# Python 3.X, 2.6+'The bright side of life' >>>'The {} side {} {}'.format('bright', 'of', 'life')# Python 3.1+, 2.7+'The bright side of life' >>>'The %s side %s %s' % ('bright', 'of', 'life')# All Pythons'The bright side of life'
>>>'{0:f}, {1:.2f}, {2:05.2f}'.format(3.14159, 3.14159, 3.14159)'3.141590, 3.14, 03.14' >>>'{:f}, {:.2f}, {:06.2f}'.format(3.14159, 3.14159, 3.14159)'3.141590, 3.14, 003.14' >>>'%f, %.2f, %06.2f' % (3.14159, 3.14159, 3.14159)'3.141590, 3.14, 003.14'
>>>'%.2f' % 1.2345# Single value'1.23' >>>'%.2f %s' % (1.2345, 99)# Multiple values tuple'1.23 99'
>>>'%s' % 1.23# Single value, by itself'1.23' >>>'%s' % (1.23,)# Single value, in a tuple'1.23' >>>'%s' % ((1.23,),)# Single value that is a tuple'(1.23,)'
>>>'{0:.2f}'.format(1.2345)# Single value'1.23' >>>'{0:.2f} {1}'.format(1.2345, 99)# Multiple values'1.23 99' >>>'{0}'.format(1.23)# Single value, by itself'1.23' >>>'{0}'.format((1.23,))# Single value that is a tuple'(1.23,)'
def myformat(fmt, args): return fmt % args# See Part IVmyformat('%s %s', (88, 99))# Call your function objectstr.format('{} {}', 88, 99)# Versus calling the built-inotherfunction(myformat)# Your function is an object too
>>>'%(num)i = %(title)s' % dict(num=7, title='Strings')'7 = Strings' >>>'{num:d} = {title:s}'.format(num=7, title='Strings')'7 = Strings' >>>'{num} = {title}'.format(**dict(num=7, title='Strings'))'7 = Strings'
>>>import string>>>t = string.Template('$num = $title')>>>t.substitute({'num': 7, 'title': 'Strings'})'7 = Strings' >>>t.substitute(num=7, title='Strings')'7 = Strings' >>>t.substitute(dict(num=7, title='Strings'))'7 = Strings'
None of the object types in the immutable category support in-place changes, though we can always run expressions to make new objects and assign their results to variables as needed.
Conversely, the mutable types can always be changed in place with operations that do not create new objects. Although such objects can be copied, in-place changes support direct modification.
find method be
used to search a list?S with the
value "s,pa,m", name two ways to
extract the two characters in the middle."a\nb\x1f\000d"?string
module instead of string method calls?X+Y and built-in functions like len(X) are generic, though, and may work on
a variety of types. In this case, for instance, the in membership expression has a similar
effect as the string find, but it
can be used to search both strings and lists. In Python 3.X, there is
some attempt to group methods by categories (for example, the mutable
sequence types list and bytearray have similar method sets), but
methods are still more type-specific than other operation sets.ord(S) function converts from a one-character
string to an integer character code; chr(I) converts from the integer code back to a
string. Keep in mind, though, that these integers are only ASCII codes
for text whose characters are drawn only from ASCII character set. In
the Unicode model, text strings are really sequences of Unicode code
point identifying integers, which may fall outside the 7-bit range of
numbers reserved by ASCII (more on Unicode in Chapter 4 and Chapter 37).replace — and then assigning the
result back to the original variable name.S[2:4], or split on the comma and index the
string using S.split(',')[1]. Try
these interactively to see for yourself."a\nb\x1f\000d" contains the characters
a, newline (\n), literal value 31 (a hex escape \x1f), literal value 0 (an octal escape \000), and d. Pass the string to the built-in len function to verify this, and print each
of its character’s ord results to
see the actual code point (identifying number) values. See Table 7-2 for more details on
escapes.string module instead of string object
method calls today — it’s deprecated, and its calls are removed
completely in Python 3.X. The only valid reason for using the string module at all today is for its other
tools, such as predefined constants. You might also see it appear in
what is now very old and dusty Python code (and books of the misty
past — like the 1990s).1 More mathematically minded readers (and students in my classes) sometimes detect a small asymmetry here: the leftmost item is at offset 0, but the rightmost is at offset −1. Alas, there is no such thing as a distinct −0 value in Python.
2 See also the Chapter 31 note
about a str.format bug (or
regression) in Pythons 3.2 and 3.3 concerning generic empty
substitution targets for object attributes that define no __format__ handler. This impacted a
working example from this book’s prior edition. While it may be a
temporary regression, it does at the least underscore that this
method is still a bit of a moving target — yet another reason to
question the feature redundancy it implies.
From a functional view, lists are just places to collect other objects so you can treat them as groups. Lists also maintain a left-to-right positional ordering among the items they contain (i.e., they are sequences).
Just as with strings, you can fetch a component object out of a list by indexing the list on the object’s offset. Because items in lists are ordered by their positions, you can also do tasks such as slicing and concatenation.
Unlike strings, lists can grow and shrink in place (their lengths can vary), and they can contain any sort of object, not just one-character strings (they’re heterogeneous). Because lists can contain other complex objects, they also support arbitrary nesting; you can create lists of lists of lists, and so on.
In terms of our type category qualifiers, lists are mutable (i.e., can be changed in place) and can respond to all the sequence operations used with strings, such as indexing, slicing, and concatenation. In fact, sequence operations work the same on lists as they do on strings; the only difference is that sequence operations such as concatenation and slicing return new lists instead of new strings when applied to lists. Because lists are mutable, however, they also support other operations that strings don’t, such as deletion and index assignment operations, which change the lists in place.
Technically, Python lists contain zero or more references to other objects. Lists might remind you of arrays of pointers (addresses) if you have a background in some other languages. Fetching an item from a Python list is about as fast as indexing a C array; in fact, lists really are arrays inside the standard Python interpreter, not linked structures. As we learned in Chapter 6, though, Python always follows a reference to an object whenever the reference is used, so your program deals only with objects. Whenever you assign an object to a data structure component or variable name, Python always stores a reference to that same object, not a copy of it (unless you request a copy explicitly).
| Operation | Interpretation |
|---|---|
%python>>>len([1, 2, 3])# Length3 >>>[1, 2, 3] + [4, 5, 6]# Concatenation[1, 2, 3, 4, 5, 6] >>>['Ni!'] * 4# Repetition['Ni!', 'Ni!', 'Ni!', 'Ni!']
>>>str([1, 2]) + "34"# Same as "[1, 2]" + "34"'[1, 2]34' >>>[1, 2] + list("34")# Same as [1, 2] + ["3", "4"][1, 2, '3', '4']
>>>3 in [1, 2, 3]# MembershipTrue >>>for x in [1, 2, 3]:...print(x, end=' ')# Iteration (2.X uses: print x,)... 1 2 3
>>>res =[c * 4 for c in 'SPAM']# List comprehensions>>>res['SSSS', 'PPPP', 'AAAA', 'MMMM']
>>>res = []>>>for c in 'SPAM':# List comprehension equivalent...res.append(c * 4)... >>>res['SSSS', 'PPPP', 'AAAA', 'MMMM']
>>>list(map(abs, [−1, −2, 0, 1, 2]))# Map a function across a sequence[1, 2, 0, 1, 2]
>>>L = ['spam', 'Spam', 'SPAM!']>>>L[2]# Offsets start at zero'SPAM!' >>>L[−2]# Negative: count from the right'Spam' >>>L[1:]# Slicing fetches sections['Spam', 'SPAM!']
>>> matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]>>>matrix[1][4, 5, 6] >>>matrix[1][1]5 >>>matrix[2][0]7 >>>matrix = [[1, 2, 3],...[4, 5, 6],...[7, 8, 9]]>>>matrix[1][1]5
>>>L = ['spam', 'Spam', 'SPAM!']>>>L[1] = 'eggs'# Index assignment>>>L['spam', 'eggs', 'SPAM!'] >>>L[0:2] = ['eat', 'more']# Slice assignment: delete+insert>>>L# Replaces items 0,1['eat', 'more', 'SPAM!']
= is
deleted.= are inserted into the list on the
left, at the place where the old slice was deleted.2>>>L = [1, 2, 3]>>>L[1:2] = [4, 5]# Replacement/insertion>>>L[1, 4, 5, 3] >>>L[1:1] = [6, 7]# Insertion (replace nothing)>>>L[1, 6, 7, 4, 5, 3] >>>L[1:2] = []# Deletion (insert nothing)>>>L[1, 7, 4, 5, 3]
>>>L = [1]>>>L[:0] = [2, 3, 4]# Insert all at :0, an empty slice at front>>>L[2, 3, 4, 1] >>>L[len(L):] = [5, 6, 7]# Insert all at len(L):, an empty slice at end>>>L[2, 3, 4, 1, 5, 6, 7] >>>L.extend([8, 9, 10])# Insert all at end, named method>>>L[2, 3, 4, 1, 5, 6, 7, 8, 9, 10]
>>>L = ['eat', 'more', 'SPAM!']>>>L.append('please')# Append method call: add item at end>>>L['eat', 'more', 'SPAM!', 'please'] >>>L.sort()# Sort list items ('S' < 'e')>>>L['SPAM!', 'eat', 'more', 'please']
>>>L = ['abc', 'ABD', 'aBe']>>>L.sort()# Sort with mixed case>>>L['ABD', 'aBe', 'abc'] >>>L = ['abc', 'ABD', 'aBe']>>>L.sort(key=str.lower)# Normalize to lowercase>>>L['abc', 'ABD', 'aBe'] >>> >>>L = ['abc', 'ABD', 'aBe']>>>L.sort(key=str.lower, reverse=True)# Change sort order>>>L['aBe', 'ABD', 'abc']
>>>L = ['abc', 'ABD', 'aBe']>>>sorted(L, key=str.lower, reverse=True)# Sorting built-in['aBe', 'ABD', 'abc'] >>>L = ['abc', 'ABD', 'aBe']>>>sorted([x.lower() for x in L], reverse=True)# Pretransform items: differs!['abe', 'abd', 'abc']
>>>L = [1, 2]>>>L.extend([3, 4, 5])# Add many items at end (like in-place +)>>>L[1, 2, 3, 4, 5] >>>L.pop()# Delete and return last item (by default: −1)5 >>>L[1, 2, 3, 4] >>>L.reverse()# In-place reversal method>>>L[4, 3, 2, 1] >>>list(reversed(L))# Reversal built-in with a result (iterator)[1, 2, 3, 4]
>>>L = []>>>L.append(1)# Push onto stack>>>L.append(2)>>>L[1, 2] >>>L.pop()# Pop off stack2 >>>L[1]
>>>L = ['spam', 'eggs', 'ham']>>>L.index('eggs')# Indexofan object (search/find)1 >>>L.insert(1, 'toast')# Insert at position>>>L['spam', 'toast', 'eggs', 'ham'] >>>L.remove('eggs')# Delete by value>>>L['spam', 'toast', 'ham'] >>>L.pop(1)# Delete by position'toast' >>>L['spam', 'ham'] >>>L.count('spam')# Number of occurrences1
>>>L = ['spam', 'eggs', 'ham', 'toast']>>>del L[0]# Delete one item>>>L['eggs', 'ham', 'toast'] >>>del L[1:]# Delete an entire section>>>L# Same as L[1:] = []['eggs']
>>>L = ['Already', 'got', 'one']>>>L[1:] = []>>>L['Already'] >>>L[0] = []>>>L[[]]
Dictionaries are sometimes called associative arrays or hashes (especially by users of other scripting languages). They associate a set of values with keys, so you can fetch an item out of a dictionary using the key under which you originally stored it. You use the same indexing operation to get components in a dictionary as you do in a list, but the index takes the form of a key, not a relative offset.
Unlike in a list, items stored in a dictionary aren’t kept in any particular order; in fact, Python pseudo-randomizes their left-to-right order to provide quick lookup. Keys provide the symbolic (not physical) locations of items in a dictionary.
Like lists, dictionaries can grow and shrink in place (without new copies being made), they can contain objects of any type, and they support nesting to any depth (they can contain lists, other dictionaries, and so on). Each key can have just one associated value, but that value can be a collection of multiple objects if needed, and a given value can be stored under any number of keys.
You can change dictionaries in place by assigning to indexes (they are mutable), but they don’t support the sequence operations that work on strings and lists. Because dictionaries are unordered collections, operations that depend on a fixed positional order (e.g., concatenation, slicing) don’t make sense. Instead, dictionaries are the only built-in, core type representatives of the mapping category — objects that map keys to values. Other mappings in Python are created by imported modules.
If lists are arrays of object references that support access by position, dictionaries are unordered tables of object references that support access by key. Internally, dictionaries are implemented as hash tables (data structures that support very fast retrieval), which start small and grow on demand. Moreover, Python employs optimized hashing algorithms to find keys, so retrieval is quick. Like lists, dictionaries store object references (not copies, unless you ask for them explicitly).
| Operation | Interpretation |
|---|---|
%python>>>D = {'spam': 2, 'ham': 1, 'eggs': 3}# Make a dictionary>>>D['spam']# Fetch a value by key2 >>>D# Order is "scrambled"{'eggs': 3, 'spam': 2, 'ham': 1}
>>>len(D)# Number of entries in dictionary3 >>>'ham' in D# Key membership test alternativeTrue >>>list(D.keys())# Create a new list of D's keys['eggs', 'spam', 'ham']
>>>D{'eggs': 3, 'spam': 2, 'ham': 1} >>>D['ham'] = ['grill', 'bake', 'fry']# Change entry (value=list)>>>D{'eggs': 3, 'spam': 2, 'ham': ['grill', 'bake', 'fry']} >>>del D['eggs']# Delete entry>>>D{'spam': 2, 'ham': ['grill', 'bake', 'fry']} >>>D['brunch'] = 'Bacon'# Add new entry>>>D{'brunch': 'Bacon', 'spam': 2, 'ham': ['grill', 'bake', 'fry']}
>>>D = {'spam': 2, 'ham': 1, 'eggs': 3}>>>list(D.values())[3, 2, 1] >>>list(D.items())[('eggs', 3), ('spam', 2), ('ham', 1)]
>>>D.get('spam')# A key that is there2 >>>print(D.get('toast'))# A key that is missingNone >>>D.get('toast', 88)88
>>>D{'eggs': 3, 'spam': 2, 'ham': 1} >>>D2 = {'toast':4, 'muffin':5}# Lots of delicious scrambled order here>>>D.update(D2)>>>D{'eggs': 3, 'muffin': 5, 'toast': 4, 'spam': 2, 'ham': 1}
# pop a dictionary by key>>>D{'eggs': 3, 'muffin': 5, 'toast': 4, 'spam': 2, 'ham': 1} >>>D.pop('muffin')5 >>>D.pop('toast')# Delete and return from a key4 >>>D{'eggs': 3, 'spam': 2, 'ham': 1}# pop a list by position>>>L = ['aa', 'bb', 'cc', 'dd']>>>L.pop()# Delete and return from the end'dd' >>>L['aa', 'bb', 'cc'] >>>L.pop(1)# Delete from a specific position'bb' >>>L['aa', 'cc']
>>>table = {'1975': 'Holy Grail',# Key: Value...'1979': 'Life of Brian',...'1983': 'The Meaning of Life'}>>> >>>year = '1983'>>>movie = table[year]# dictionary[Key] => Value>>>movie'The Meaning of Life' >>>for year in table:# Same as: for year in table.keys()...print(year + '\t' + table[year])... 1979 Life of Brian 1975 Holy Grail 1983 The Meaning of Life
>>>table = {'Holy Grail': '1975',# Key=>Value (title=>year)...'Life of Brian': '1979',...'The Meaning of Life': '1983'}>>> >>>table['Holy Grail']'1975' >>>list(table.items())# Value=>Key (year=>title)[('The Meaning of Life', '1983'), ('Holy Grail', '1975'), ('Life of Brian', '1979')] >>>[title for (title, year) in table.items() if year == '1975']['Holy Grail']
>>>K = 'Holy Grail'>>>table[K]# Key=>Value (normal usage)'1975' >>>V = '1975'>>>[key for (key, value) in table.items() if value == V]# Value=>Key['Holy Grail'] >>>[key for key in table.keys() if table[key] == V]# Ditto['Holy Grail']
>>>L = []>>>L[99] = 'spam'Traceback (most recent call last): File "<stdin>", line 1, in ? IndexError: list assignment index out of range
>>>D = {}>>>D[99] = 'spam'>>>D[99]'spam' >>>D{99: 'spam'}
>>>table = {1975: 'Holy Grail',...1979: 'Life of Brian',# Keys are integers, not strings...1983: 'The Meaning of Life'}>>>table[1975]'Holy Grail' >>>list(table.items())[(1979, 'Life of Brian'), (1983, 'The Meaning of Life'), (1975, 'Holy Grail')]
>>>Matrix = {}>>>Matrix[(2, 3, 4)] = 88>>>Matrix[(7, 8, 9)] = 99>>> >>>X = 2; Y = 3; Z = 4# ; separates statements: see Chapter 10>>>Matrix[(X, Y, Z)]88 >>>Matrix{(2, 3, 4): 88, (7, 8, 9): 99}
>>> Matrix[(2,3,6)]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
KeyError: (2, 3, 6)>>>if (2, 3, 6) in Matrix:# Check for key before fetch...print(Matrix[(2, 3, 6)])# See Chapters 10 and 12 for if/else...else:...print(0)... 0 >>>try:...print(Matrix[(2, 3, 6)])# Try to index...except KeyError:# Catch and recover...print(0)# See Chapters 10 and 34 for try/except... 0 >>>Matrix.get((2, 3, 4), 0)# Exists: fetch and return88 >>>Matrix.get((2, 3, 6), 0)# Doesn't exist: use default arg0
>>>rec = {}>>>rec['name'] = 'Bob'>>>rec['age'] = 40.5>>>rec['job'] = 'developer/manager'>>> >>>print(rec['name'])Bob
>>>rec = {'name': 'Bob',...'jobs': ['developer', 'manager'],...'web': 'www.bobs.org/˜Bob',...'home': {'state': 'Overworked', 'zip': 12345}}
>>>rec['name']'Bob' >>>rec['jobs']['developer', 'manager'] >>>rec['jobs'][1]'manager' >>>rec['home']['zip']12345
db = [] db.append(rec)# A list "database"db.append(other) db[0]['jobs'] db = {} db['bob'] = rec# A dictionary "database"db['sue'] =otherdb['bob']['jobs']
{'name': 'Bob', 'age': 40} # Traditional literal expression
D = {} # Assign by keys dynamically
D['name'] = 'Bob'
D['age'] = 40
dict(name='Bob', age=40) # dict keyword argument form
dict([('name', 'Bob'), ('age', 40)]) # dict key/value tuples formdict(zip(keyslist, valueslist)) # Zipped key/value tuples form (ahead)>>> dict.fromkeys(['a', 'b'], 0)
{'a': 0, 'b': 0}D.keys,
D.values, and D.itemsD.has_key
method — the in membership test is
used insteadD.viewkeys,
D.viewvalues, and D.viewitems; their nonview methods return
lists as before>>>list(zip(['a', 'b', 'c'], [1, 2, 3]))# Zip together keys and values[('a', 1), ('b', 2), ('c', 3)] >>>D = dict(zip(['a', 'b', 'c'], [1, 2, 3]))# Make a dict from zip result>>>D{'b': 2, 'c': 3, 'a': 1}
>>>D = {k: v for (k, v) in zip(['a', 'b', 'c'], [1, 2, 3])}>>>D{'b': 2, 'c': 3, 'a': 1}
>>>D = {x: x ** 2 for x in [1, 2, 3, 4]}# Or: range(1, 5)>>>D{1: 1, 2: 4, 3: 9, 4: 16} >>>D = {c: c * 4 for c in 'SPAM'}# Loop over any iterable>>>D{'S': 'SSSS', 'P': 'PPPP', 'A': 'AAAA', 'M': 'MMMM'} >>>D = {c.lower(): c + '!' for c in ['SPAM', 'EGGS', 'HAM']}>>>D{'eggs': 'EGGS!', 'spam': 'SPAM!', 'ham': 'HAM!'}
>>>D = dict.fromkeys(['a', 'b', 'c'], 0)# Initialize dict from keys>>>D{'b': 0, 'c': 0, 'a': 0} >>>D = {k:0 for k in ['a', 'b', 'c']}# Same, but with a comprehension>>>D{'b': 0, 'c': 0, 'a': 0} >>>D = dict.fromkeys('spam')# Other iterables, default value>>>D{'s': None, 'p': None, 'a': None, 'm': None} >>>D = {k: None for k in 'spam'}>>>D{'s': None, 'p': None, 'a': None, 'm': None}
>>>D = dict(a=1, b=2, c=3)>>>D{'b': 2, 'c': 3, 'a': 1} >>>K = D.keys()# Makes a view object in 3.X, not a list>>>Kdict_keys(['b', 'c', 'a']) >>>list(K)# Force a real list in 3.X if needed['b', 'c', 'a'] >>>V = D.values()# Ditto for values and items views>>>Vdict_values([2, 3, 1]) >>>list(V)[2, 3, 1] >>>D.items()dict_items([('b', 2), ('c', 3), ('a', 1)]) >>>list(D.items())[('b', 2), ('c', 3), ('a', 1)] >>>K[0]# List operations fail unless convertedTypeError: 'dict_keys' object does not support indexing >>>list(K)[0]'b'
>>>for k in D.keys(): print(k)# Iterators used automatically in loops... b c a
>>>for key in D: print(key)# Still no need to call keys() to iterate... b c a
>>>D = {'a': 1, 'b': 2, 'c': 3}>>>D{'b': 2, 'c': 3, 'a': 1} >>>K = D.keys()>>>V = D.values()>>>list(K)# Views maintain same order as dictionary['b', 'c', 'a'] >>>list(V)[2, 3, 1] >>>del D['b']# Change the dictionary in place>>>D{'c': 3, 'a': 1} >>>list(K)# Reflected in any current view objects['c', 'a'] >>>list(V)# Not true in 2.X! - lists detached from dict[3, 1]
>>>K, V(dict_keys(['c', 'a']), dict_values([3, 1])) >>>K | {'x': 4}# Keys (and some items) views are set-like{'c', 'x', 'a'} >>>V & {'x': 4}TypeError: unsupported operand type(s) for &: 'dict_values' and 'dict' >>>V & {'x': 4}.values()TypeError: unsupported operand type(s) for &: 'dict_values' and 'dict_values'
>>>D = {'a': 1, 'b': 2, 'c': 3}>>>D.keys() & D.keys()# Intersect keys views{'b', 'c', 'a'} >>>D.keys() & {'b'}# Intersect keys and set{'b'} >>>D.keys() & {'b': 1}# Intersect keys and dict{'b'} >>>D.keys() | {'b', 'c', 'd'}# Union keys and set{'b', 'c', 'a', 'd'}
>>>D = {'a': 1}>>>list(D.items())# Items set-like if hashable[('a', 1)] >>>D.items() | D.keys()# Union view and view{('a', 1), 'a'} >>>D.items() | D# dict treated same as its keys{('a', 1), 'a'} >>>D.items() | {('c', 3), ('d', 4)}# Set of key/value pairs{('d', 4), ('a', 1), ('c', 3)} >>>dict(D.items() | {('c', 3), ('d', 4)})# dict accepts iterable sets too{'c': 3, 'a': 1, 'd': 4}
>>>D = {'a': 1, 'b': 2, 'c': 3}>>>D{'b': 2, 'c': 3, 'a': 1} >>>Ks = D.keys()# Sorting a view object doesn't work!>>>Ks.sort()AttributeError: 'dict_keys' object has no attribute 'sort'
>>>Ks = list(Ks)# Force it to be a list and then sort>>>Ks.sort()>>>for k in Ks: print(k, D[k])# 2.X: omit outer parens in prints... a 1 b 2 c 3 >>>D{'b': 2, 'c': 3, 'a': 1} >>>Ks = D.keys()# Or you can use sorted() on the keys>>>for k in sorted(Ks): print(k, D[k])# sorted() accepts any iterable...# sorted() returns its resulta 1 b 2 c 3
>>>D{'b': 2, 'c': 3, 'a': 1}# Better yet, sort the dict directly>>>for k in sorted(D): print(k, D[k])# dict iterators return keys... a 1 b 2 c 3
sorted(D1.items()) < sorted(D2.items()) # Like 2.X D1 < D2>>>D{'b': 2, 'c': 3, 'a': 1} >>>D.has_key('c')# 2.X only: True/FalseAttributeError: 'dict' object has no attribute 'has_key' >>>'c' in D# Required in 3.XTrue >>>'x' in D# Preferred in 2.X todayFalse >>>if 'c' in D: print('present', D['c'])# Branch on result... present 3 >>>print(D.get('c'))# Fetch with default3 >>>print(D.get('x'))None >>>if D.get('c') != None: print('present', D['c'])# Another option... present 3
'a' and 'b', each having an associated value of
0.[0, 0, 0, 0,
0] and a repetition expression like [0] * 5 will each create a list of five
zeros. In practice, you might also build one up with a loop that
starts with an empty list and appends 0 to it in each iteration, with L.append(0). A list comprehension ([0 for i in range(5)]) could work here, too,
but this is more work than you need to do for this answer.{'a': 0,
'b': 0} or a series of assignments like D = {}, D['a'] =
0, and D['b'] = 0 would
create the desired dictionary. You can also use the newer and
simpler-to-code dict(a=0, b=0)
keyword form, or the more flexible dict([('a', 0), ('b', 0)]) key/value
sequences form. Or, because all the values are the same, you can use
the special form dict.fromkeys('ab',
0). In 3.X and 2.7, you can also use a dictionary
comprehension: {k:0 for k in 'ab'},
though again, this may be overkill here.append and extend methods grow a list in place, the
sort and reverse methods order and reverse lists, the
insert method inserts an item at an
offset, the remove and pop methods delete from a list by value and
by position, the del statement
deletes an item or slice, and index and slice assignment statements
replace an item or entire section. Pick any four of these for the
quiz.del statement deletes a
key’s entry, the dictionary update
method merges one dictionary into another in place, and D.pop(key) removes a key and returns the
value it had. Dictionaries also have other, more exotic in-place
change methods not presented in this chapter, such as setdefault; see reference sources for more
details.1 In practice, you won’t see many lists written out like this in list-processing programs. It’s more common to see code that processes lists constructed dynamically (at runtime), from user inputs, file contents, and so on. In fact, although it’s important to master literal syntax, many data structures in Python are built by running program code at runtime.
2 This description requires elaboration when the value and
the slice being assigned overlap: L[2:5]=L[3:6], for instance, works
fine because the value to be inserted is fetched before the
deletion happens on the left.
3 Unlike + concatenation,
append doesn’t have to generate
new objects, so it’s usually faster than + too. You can also mimic append with the clever slice assignments
of the prior section: L[len(L):]=[X] is like L.append(X), and L[:0]=[X] is like appending at the front
of a list. Both delete an empty slice and insert X, changing L in place quickly, like append. Both are arguably more complex
than list methods, though. For instance, L.insert(0, X) can also append an item
to the front of a list, and seems noticeably more mnemonic;
L.insert(len(L), X) inserts one
object at the end too, but unless you like typing, you might as
well use L.append(X)!
4 As for lists, you might not see dictionaries coded in full using
literals very often — programs rarely know all their data before they
are run, and more typically extract it dynamically from users, files,
and so on. Lists and dictionaries are grown in different ways, though.
In the next section you’ll see that you often build up dictionaries by
assigning to new keys at runtime; this approach fails for lists, which
are commonly grown with append or
extend instead.
Like strings and lists, tuples are positionally ordered collections of objects (i.e., they maintain a left-to-right order among their contents); like lists, they can embed any kind of object.
Like strings and lists, items in a tuple are accessed by offset (not by key); they support all the offset-based access operations, such as indexing and slicing.
Like strings and lists, tuples are sequences; they support many of the same operations. However, like strings, tuples are immutable; they don’t support any of the in-place change operations applied to lists.
Because tuples are immutable, you cannot change the size of a tuple without making a copy. On the other hand, tuples can hold any type of object, including other compound objects (e.g., lists, dictionaries, other tuples), and so support arbitrary nesting.
Like lists, tuples are best thought of as object reference arrays; tuples store access points to other objects (references), and indexing a tuple is relatively quick.
| Operation | Interpretation |
|---|---|
>>>(1, 2) + (3, 4)# Concatenation(1, 2, 3, 4) >>>(1, 2) * 4# Repetition(1, 2, 1, 2, 1, 2, 1, 2) >>>T = (1, 2, 3, 4)# Indexing, slicing>>>T[0], T[1:3](1, (2, 3))
>>>x = (40)# An integer!>>>x40 >>>y = (40,)# A tuple containing an integer>>>y(40,)
print
statement.>>>T = ('cc', 'aa', 'dd', 'bb')>>>tmp = list(T)# Make a list from a tuple's items>>>tmp.sort()# Sort the list>>>tmp['aa', 'bb', 'cc', 'dd'] >>>T = tuple(tmp)# Make a tuple from the list's items>>>T('aa', 'bb', 'cc', 'dd') >>>sorted(T)# Or use the sorted built-in, and save two steps['aa', 'bb', 'cc', 'dd']
>>>T = (1, 2, 3, 4, 5)>>>L = [x + 20 for x in T]>>>L[21, 22, 23, 24, 25]
>>>T = (1, 2, 3, 2, 4, 2)# Tuple methods in 2.6, 3.0, and later>>>T.index(2)# Offset of first appearance of 21 >>>T.index(2, 2)# Offset of appearance after offset 23 >>>T.count(2)# How many 2s are there?3
>>>T = (1, [2, 3], 4)>>>T[1] = 'spam'# This fails: can't change tuple itselfTypeError: object doesn't support item assignment >>>T[1][0] = 'spam'# This works: can change mutables inside>>>T(1, ['spam', 3], 4)
>>>bob = ('Bob', 40.5, ['dev', 'mgr'])# Tuple record>>>bob('Bob', 40.5, ['dev', 'mgr']) >>>bob[0], bob[2]# Access by position('Bob', ['dev', 'mgr'])
>>>bob = dict(name='Bob', age=40.5, jobs=['dev', 'mgr'])# Dictionary record>>>bob{'jobs': ['dev', 'mgr'], 'name': 'Bob', 'age': 40.5} >>>bob['name'], bob['jobs']# Access by key('Bob', ['dev', 'mgr'])
>>>tuple(bob.values())# Values to tuple(['dev', 'mgr'], 'Bob', 40.5) >>>list(bob.items())# Items to tuple list[('jobs', ['dev', 'mgr']), ('name', 'Bob'), ('age', 40.5)]
>>>from collections import namedtuple# Import extension type>>>Rec = namedtuple('Rec', ['name', 'age', 'jobs'])# Make a generated class>>>bob = Rec('Bob', age=40.5, jobs=['dev', 'mgr'])# A named-tuple record>>>bobRec(name='Bob', age=40.5, jobs=['dev', 'mgr']) >>>bob[0], bob[2]# Access by position('Bob', ['dev', 'mgr']) >>>bob.name, bob.jobs# Access by attribute('Bob', ['dev', 'mgr'])
>>>O = bob._asdict()# Dictionary-like form>>>O['name'], O['jobs']# Access by key too('Bob', ['dev', 'mgr']) >>>OOrderedDict([('name', 'Bob'), ('age', 40.5), ('jobs', ['dev', 'mgr'])])
>>>bob = Rec('Bob', 40.5, ['dev', 'mgr'])# For both tuples and named tuples>>>name, age, jobs = bob# Tuple assignment (Chapter 11)>>>name, jobs('Bob', ['dev', 'mgr']) >>>for x in bob: print(x)# Iteration context (Chapters 14, 20)...prints Bob, 40.5, ['dev', 'mgr']...
>>>bob = {'name': 'Bob', 'age': 40.5, 'jobs': ['dev', 'mgr']}>>>job, name, age = bob.values()>>>name, job# Dict equivalent (but order may vary)('Bob', ['dev', 'mgr']) >>>for x in bob: print(bob[x])# Step though keys, index values...prints values...>>>for x in bob.values(): print(x)# Step through values view...prints values...
| Operation | Interpretation |
|---|---|
afile = open(filename,mode) afile.method()
b to the mode
string allows for binary data (end-of-line
translations and 3.X Unicode encodings are turned off).+ opens the file
for both input and output (i.e., you can both
read and write to the same file object, often in conjunction with
seek operations to reposition in the file).Though the reading and writing methods in the table are common, keep in mind that probably the best way to read lines from a text file today is to not read the file at all — as we’ll see in Chapter 14, files also have an iterator that automatically reads one line at a time in aforloop, list comprehension, or other iteration context.
Notice in Table 9-2 that data read from a file always comes back to your script as a string, so you’ll have to convert it to a different type of Python object if a string is not what you need. Similarly, unlike with theint,float,str, and the string formatting expression and method) come in handy when dealing with files.Python also includes advanced standard library tools for handling generic object storage (thepicklemodule), for dealing with packed binary data in files (thestructmodule), and for processing special types of content such as JSON, XML, and CSV text. We’ll see these at work later in this chapter and book, but Python’s manuals document them in full.
By default, output files are always buffered, which means that text you write may not be transferred from memory to disk immediately — closing a file, or running itsflushmethod, forces the buffered data to disk. You can avoid buffering with extraopenarguments, but it may impede performance. Python files are also random-access on a byte offset basis — theirseekmethod allows your scripts to jump around to read and write at specific locations.
close is often optional:
auto-close on collectionCalling the fileclosemethod terminates your connection to the external file, releases its system resources, and flushes its buffered output to disk if any is still in memory. As discussed in Chapter 6, in Python an object’s memory space is automatically reclaimed as soon as the object is no longer referenced anywhere in the program. When file objects are reclaimed, Python also automatically closes the files if they are still open (this also happens when a program shuts down). This means you don’t always need to manually close your files in standard Python, especially those in simple scripts with short runtimes, and temporary files used by a single line or expression.On the other hand, including manualclosecalls doesn’t hurt, and may be a good habit to form, especially in long-running systems. Strictly speaking, this auto-close-on-collection feature of files is not part of the language definition — it may change over time, may not happen when you expect it to in interactive shells, and may not work the same in other Python implementations whose garbage collectors may not reclaim and close files at the same points as standard CPython. In fact, when many files are opened within loops, Pythons other than CPython may require close calls to free up system resources immediately, before garbage collection can get around to freeing objects. Moreover, close calls may sometimes be required to flush buffered output of file objects not yet reclaimed. For an alternative way to guarantee automatic file closes, also see this section’s later discussion of the file object’s context manager, used with thewith/asstatement in Python 2.6, 2.7, and 3.X.
>>>myfile = open('myfile.txt', 'w')# Open for text output: create/empty>>>myfile.write('hello text file\n')# Write a line of text: string16 >>>myfile.write('goodbye text file\n')18 >>>myfile.close()# Flush output buffers to disk>>>myfile = open('myfile.txt')# Open for text input: 'r' is default>>>myfile.readline()# Read the lines back'hello text file\n' >>>myfile.readline()'goodbye text file\n' >>>myfile.readline()# Empty string: end-of-file''
>>>open('myfile.txt').read()# Read all at once into string'hello text file\ngoodbye text file\n' >>>print(open('myfile.txt').read())# User-friendly displayhello text file goodbye text file
>>>for line in open('myfile.txt'):# Use file iterators, not reads...print(line, end='')... hello text file goodbye text file
>>>open(r'C:\Python33\Lib\pdb.py').readline()'#! /usr/bin/env python3\n' >>>open('C:/Python33/Lib/pdb.py').readline()'#! /usr/bin/env python3\n' >>>open('C:\\Python33\\Lib\\pdb.py').readline()'#! /usr/bin/env python3\n'
str strings,
perform Unicode encoding and decoding automatically, and perform
end-of-line translation by default.bytes
string type and allow programs to access file content
unaltered.>>>data = open('data.bin', 'rb').read()# Open binary file: rb=read binary>>>data# bytes string holds binary datab'\x00\x00\x00\x07spam\x00\x08' >>>data[4:8]# Act like stringsb'spam' >>>data[4:8][0]# But really are small 8-bit integers115 >>>bin(data[4:8][0])# Python 3.X/2.6+ bin() function'0b1110011'
>>>X, Y, Z = 43, 44, 45# Native Python objects>>>S = 'Spam'# Must be strings to store in file>>>D = {'a': 1, 'b': 2}>>>L = [1, 2, 3]>>> >>>F = open('datafile.txt', 'w')# Create output text file>>>F.write(S + '\n')# Terminate lines with \n>>>F.write('%s,%s,%s\n' % (X, Y, Z))# Convert numbers to strings>>>F.write(str(L) + '$' + str(D) + '\n')# Convert and separate with $>>>F.close()
>>>chars = open('datafile.txt').read()# Raw string display>>>chars"Spam\n43,44,45\n[1, 2, 3]${'a': 1, 'b': 2}\n" >>>print(chars)# User-friendly displaySpam 43,44,45 [1, 2, 3]${'a': 1, 'b': 2}
>>>F = open('datafile.txt')# Open again>>>line = F.readline()# Read one line>>>line'Spam\n' >>>line.rstrip()# Remove end-of-line'Spam'
>>>line = F.readline()# Next line from file>>>line# It's a string here'43,44,45\n' >>>parts = line.split(',')# Split (parse) on commas>>>parts['43', '44', '45\n']
>>>int(parts[1])# Convert from string to int44 >>>numbers = [int(P) for P in parts]# Convert all in list at once>>>numbers[43, 44, 45]
>>>line = F.readline()>>>line"[1, 2, 3]${'a': 1, 'b': 2}\n" >>>parts = line.split('$')# Split (parse) on $>>>parts['[1, 2, 3]', "{'a': 1, 'b': 2}\n"] >>>eval(parts[0])# Convert to any object type[1, 2, 3] >>>objects = [eval(P) for P in parts]# Do same for all in list>>>objects[[1, 2, 3], {'a': 1, 'b': 2}]
>>>D = {'a': 1, 'b': 2}>>>F = open('datafile.pkl', 'wb')>>>import pickle>>>pickle.dump(D, F)# Pickle any object to file>>>F.close()
>>>F = open('datafile.pkl', 'rb')>>>E = pickle.load(F)# Load any object from file>>>E{'a': 1, 'b': 2}
>>>open('datafile.pkl', 'rb').read()# Format is prone to change!b'\x80\x03}q\x00(X\x01\x00\x00\x00bq\x01K\x02X\x01\x00\x00\x00aq\x02K\x01u.'
>>>name = dict(first='Bob', last='Smith')>>>rec = dict(name=name, job=['dev', 'mgr'], age=40.5)>>>rec{'job': ['dev', 'mgr'], 'name': {'last': 'Smith', 'first': 'Bob'}, 'age': 40.5}
>>>import json>>>json.dumps(rec)'{"job": ["dev", "mgr"], "name": {"last": "Smith", "first": "Bob"}, "age": 40.5}' >>>S = json.dumps(rec)>>>S'{"job": ["dev", "mgr"], "name": {"last": "Smith", "first": "Bob"}, "age": 40.5}' >>>O = json.loads(S)>>>O{'job': ['dev', 'mgr'], 'name': {'last': 'Smith', 'first': 'Bob'}, 'age': 40.5} >>>O == recTrue
>>>json.dump(rec, fp=open('testjson.txt', 'w'), indent=4)>>>print(open('testjson.txt').read()){ "job": [ "dev", "mgr" ], "name": { "last": "Smith", "first": "Bob" }, "age": 40.5 } >>>P = json.load(open('testjson.txt'))>>>P{'job': ['dev', 'mgr'], 'name': {'last': 'Smith', 'first': 'Bob'}, 'age': 40.5}
>>>import csv>>>rdr = csv.reader(open('csvdata.txt'))>>>for row in rdr: print(row)... ['a', 'bbb', 'cc', 'dddd'] ['11', '22', '33', '44']
>>>F = open('data.bin', 'wb')# Open binary output file>>>import struct>>>data = struct.pack('>i4sh', 7, b'spam', 8)# Make packed binary data>>>datab'\x00\x00\x00\x07spam\x00\x08' >>>F.write(data)# Write byte string>>>F.close()
>>>F = open('data.bin', 'rb')>>>data = F.read()# Get packed binary data>>>datab'\x00\x00\x00\x07spam\x00\x08' >>>values = struct.unpack('>i4sh', data)# Convert to Python objects>>>values(7, b'spam', 8)
with open(r'C:\code\data.txt') as myfile:# See Chapter 34 for detailsfor line in myfile:...use line here...
myfile = open(r'C:\code\data.txt')
try:
for line in myfile:
...use line here...
finally:
myfile.close()Preopened file objects in thesysmodule, such assys.stdout(see “Print Operations” in Chapter 11 for details)
os
moduleFile-like objects used to synchronize processes or communicate over networks
Tools such asos.popenandsubprocess.Popenthat support spawning shell commands and reading and writing to their standard streams (see Chapter 13 and Chapter 21 for examples)
str, as well as bytes in 3.X and unicode in 2.X; the bytearray string type in 3.X, 2.6, and 2.7
is mutable.frozenset is an immutable variant of set.| Object type | Category | Mutable? |
|---|---|---|
>>>L = ['abc', [(1, 2), ([3], 4)], 5]>>>L[1][(1, 2), ([3], 4)] >>>L[1][1]([3], 4) >>>L[1][1][0][3] >>>L[1][1][0][0]3
>>>X = [1, 2, 3]>>>L = ['a', X, 'b']# Embed references to X's object>>>D = {'x':X, 'y':2}
>>>X[1] = 'surprise'# Changes all three references!>>>L['a', [1, 'surprise', 3], 'b'] >>>D{'x': [1, 'surprise', 3], 'y': 2}
L[:]) copy sequences.copy
method (X.copy()) copies a
dictionary, set, or list (the list’s copy is new as of 3.3).list and dict make copies (list(L), dict(D), set(S)).copy standard library
module makes full copies when needed.>>>L = [1,2,3]>>>D = {'a':1, 'b':2}
>>>A = L[:]# Instead of A = L (or list(L))>>>B = D.copy()# Instead of B = D (ditto for sets)
>>>A[1] = 'Ni'>>>B['c'] = 'spam'>>> >>>L, D([1, 2, 3], {'a': 1, 'b': 2}) >>>A, B([1, 'Ni', 3], {'a': 1, 'c': 'spam', 'b': 2})
>>>X = [1, 2, 3]>>>L = ['a', X[:], 'b']# Embed copies of X's object>>>D = {'x':X[:], 'y':2}
import copy
X = copy.deepcopy(Y) # Fully copy an arbitrarily nested object Y>>>L1 = [1, ('a', 3)]# Same value, unique objects>>>L2 = [1, ('a', 3)]>>>L1 == L2, L1 is L2# Equivalent? Same object?(True, False)
== operator
tests value equivalence. Python performs an equivalence test, comparing all nested objects
recursively.is operator
tests object identity. Python tests whether the two are really the same object (i.e., live at the
same address in memory).>>>S1 = 'spam'>>>S2 = 'spam'>>>S1 == S2, S1 is S2(True, True)
>>>S1 = 'a longer string'>>>S2 = 'a longer string'>>>S1 == S2, S1 is S2(True, False)
>>>L1 = [1, ('a', 3)]>>>L2 = [1, ('a', 2)]>>>L1 < L2, L1 == L2, L1 > L2# Less, equal, greater: tuple of results(False, False, True)
ord), and
character by character until the end or first mismatch ("abc" < "ac").[2] > [1, 2]).(key, value) lists are equal. Relative magnitude
comparisons are not supported for dictionaries in Python 3.X, but
they work in 2.X as though comparing sorted (key, value) lists.1 < 'spam') are errors in Python 3.X.
They are allowed in Python 2.X, but use a fixed but arbitrary
ordering rule based on type name string. By proxy, this also applies
to sorts, which use comparisons internally: nonnumeric mixed-type
collections cannot be sorted in 3.X.c:\code>c:\python27\python>>>11 == '11'# Equality does not convert non-numbersFalse >>>11 >= '11'# 2.X compares by type name string: int, strFalse >>>['11', '22'].sort()# Ditto for sorts>>>[11, '11'].sort()
c:\code>c:\python33\python>>>11 == '11'# 3.X: equality works but magnitude does notFalse >>>11 >= '11'TypeError: unorderable types: int() > str() >>>['11', '22'].sort()# Ditto for sorts>>>[11, '11'].sort()TypeError: unorderable types: str() < int() >>>11 > 9.123# Mixed numbers convert to highest typeTrue >>>str(11) >= '11', 11 >= int('11')# Manual conversions force the issue(True, True)
C:\code>c:\python27\python>>>D1 = {'a':1, 'b':2}>>>D2 = {'a':1, 'b':3}>>>D1 == D2# Dictionary equality: 2.X + 3.XFalse >>>D1 < D2# Dictionary magnitude: 2.X onlyTrue
C:\code>c:\python33\python>>>D1 = {'a':1, 'b':2}>>>D2 = {'a':1, 'b':3}>>>D1 == D2False >>>D1 < D2TypeError: unorderable types: dict() < dict()
>>>list(D1.items())[('b', 2), ('a', 1)] >>>sorted(D1.items())[('a', 1), ('b', 2)] >>> >>>sorted(D1.items()) < sorted(D2.items())# Magnitude test in 3.XTrue >>>sorted(D1.items()) > sorted(D2.items())False
| Object | Value |
|---|---|
>>>L = [None] * 100>>> >>>L[None, None, None, None, None, None, None, ... ]
True and False are equivalent to 1 and 0, but they make the programmer’s intent
clearer.True and False, instead of as 1 and 0, to make the type of result
clearer.>>>bool(1)True >>>bool('spam')True >>>bool({})False
type([1]) == type([])# Compare to type of another listtype([1]) == list# Compare to list type nameisinstance([1], list)# Test if list or customization thereofimport types# types has names for other typesdef f(): pass type(f) == types.FunctionType
>>>L = [1, 2, 3]>>>M = ['X', L, 'Y']# Embed a reference to L>>>M['X', [1, 2, 3], 'Y'] >>>L[1] = 0# Changes M too>>>M['X', [1, 0, 3], 'Y']
>>>L = [1, 2, 3]>>>M = ['X', L[:], 'Y']# Embed a copy of L (or list(L), or L.copy())>>>L[1] = 0# Changes only L, not M>>>L[1, 0, 3] >>>M['X', [1, 2, 3], 'Y']
>>>L = [4, 5, 6]>>>X = L * 4# Like [4, 5, 6] + [4, 5, 6] + ...>>>Y = [L] * 4# [L] + [L] + ... = [L, L,...]>>>X[4, 5, 6, 4, 5, 6, 4, 5, 6, 4, 5, 6] >>>Y[[4, 5, 6], [4, 5, 6], [4, 5, 6], [4, 5, 6]]
>>>L[1] = 0# Impacts Y but not X>>>X[4, 5, 6, 4, 5, 6, 4, 5, 6, 4, 5, 6] >>>Y[[4, 0, 6], [4, 0, 6], [4, 0, 6], [4, 0, 6]]
>>>L = [4, 5, 6]>>>Y = [list(L)] * 4# Embed a (shared) copy of L>>>L[1] = 0>>>Y[[4, 5, 6], [4, 5, 6], [4, 5, 6], [4, 5, 6]]
>>>Y[0][1] = 99# All four copies are still the same>>>Y[[4, 99, 6], [4, 99, 6], [4, 99, 6], [4, 99, 6]] >>>L = [4, 5, 6]>>>Y = [list(L) for i in range(4)]>>>Y[[4, 5, 6], [4, 5, 6], [4, 5, 6], [4, 5, 6]] >>>Y[0][1] = 99>>>Y[[4, 99, 6], [4, 5, 6], [4, 5, 6], [4, 5, 6]]
>>>L = ['grail']# Append reference to same object>>>L.append(L)# Generates cycle in object: [...]>>>L['grail', [...]]
T = (1, 2, 3) T[2] = 4# Error!T = T[:2] + (4,)# OK: (1, 2, 4)
(4, 5, 6) should become (1, 5, 6) in the process.open call?len function
returns the length (number of contained items) for any container
object in Python, including tuples. It is a built-in function instead
of a type method because it applies to many different types of
objects. In general, built-in functions and expressions may span many
object types; methods are specific to a single object type, though
some may be available on more than one type (index, for example, works on lists and
tuples).T = (4, 5, 6), you can change the
first item by making a new tuple from its parts by slicing and
concatenating: T = (1,) + T[1:].
(Recall that single-item tuples require a trailing comma.) You could
also convert the tuple to a list, change it in place, and convert it
back to a tuple, but this is more expensive and is rarely required in
practice — simply use a list if you know that the object will require
in-place changes.open call is 'r', for reading text input. For input text
files, simply pass in the external file’s name.pickle module can be used
to store Python objects in a file without explicitly converting them
to strings. The struct module is
related, but it assumes the data is to be in packed binary format in
the file; json similarly converts a
limited set of Python objects to and from strings per the JSON
format.copy module, and
call copy.deepcopy(X) if you need
to copy all parts of a nested structure X. This is also rarely seen in practice;
references are usually the desired behavior, and shallow copies (e.g.,
aList[:], aDict.copy(), set(aSet)) usually suffice for most
copies.True and False are essentially predefined to have the
same meanings as integer 1 and
0, respectively.X=1;X assigns and then prints a variable
(more on statement syntax in the next part of the book). Also remember
that a comma between expressions usually builds a tuple, even if there
are no enclosing parentheses: X,Y,Z
is a three-item tuple, which Python prints back to you in
parentheses.2 ** 16
2 / 5, 2 / 5.0
"spam" + "eggs"
S = "ham"
"eggs " + S
S * 5
S[:0]
"green %s and %s" % ("eggs", S)
'green {0} and {1}'.format('eggs', S)
('x',)[0]
('x', 'y')[1]
L = [1,2,3] + [4,5,6]
L, L[:], L[:0], L[−2], L[−2:]
([1,2,3] + [4,5,6])[2:4]
[L[2], L[3]]
L.reverse(); L
L.sort(); L
L.index(4)
{'a':1, 'b':2}['b']
D = {'x':1, 'y':2, 'z':3}
D['w'] = 0
D['x'] + D['w']
D[(1,2,3)] = 4
list(D.keys()), list(D.values()), (1,2,3) in D
[[]], ["",[],(),{},None]L that
contains four strings or numbers (e.g., L=[0,1,2,3]). Then, experiment with the
following boundary cases. You may never see these cases in real
programs (especially not in the bizarre ways they appear here!), but
they are intended to make you think about the underlying model, and
some may be useful in less artificial forms — slicing out of bounds can
help, for example, if a sequence is as long as you expect:L[4])?L[−1000:100])?L[3:1])? Hint: try
assigning to this slice (L[3:1]=['?']), and see where the value
is put. Do you think this may be the same phenomenon you saw when
slicing out of bounds?del. Define another list L with four items, and assign an empty list
to one of its offsets (e.g., L[2]=[]). What happens? Then, assign an
empty list to a slice (L[2:3]=[]).
What happens now? Recall that slice assignment deletes the slice and
inserts the new value where it used to be.del statement deletes
offsets, keys, attributes, and names. Use it on your list to delete an
item (e.g., del L[0]). What happens
if you delete an entire slice (del
L[1:])? What happens when you assign a nonsequence to a
slice (L[1:2]=1)?>>>X = 'spam'>>>Y = 'eggs'>>>X, Y = Y, X
X and Y
when you type this sequence?>>>D = {}>>>D[1] = 'a'>>>D[2] = 'b'
>>>D[(1, 2, 3)] = 'c'>>>D{1: 'a', 2: 'b', (1, 2, 3): 'c'}
D with three entries, for
keys 'a', 'b', and 'c'. What happens if you try to index a
nonexistent key (D['d'])? What does
Python do if you try to assign to a nonexistent key 'd' (e.g., D['d']='spam')? How does this compare to
out-of-bounds assignments and references for lists? Does this sound
like the rule for variable names?+ operator on different/mixed types
(e.g., string + list, list
+ tuple)?+ work when one of
the operands is a dictionary?append method
work for both lists and strings? How about using the keys method on lists? (Hint: what does
append assume about its subject
object?)S of four characters: S = "spam". Then type the following
expression: S[0][0][0][0][0]. Any
clue as to what’s happening this time? (Hint: recall that a string is
a collection of characters, but Python characters are one-character
strings.) Does this indexing expression still work if you apply it to
a list such as ['s', 'p', 'a',
'm']? Why?S of four characters again: S = "spam". Write an assignment that changes
the string to "slam", using only
slicing and concatenation. Could you perform the same operation using
just indexing and concatenation? How about index assignment?"Hello file
world!" into it. Then write another script that opens
myfile.txt and reads and prints
its contents. Run your two scripts from the system command line. Does
the new file show up in the directory where you ran your scripts? What
if you add a different directory path to the filename passed to
open? Note: file write methods do not add newline characters
to your strings; add an explicit \n
at the end of the string if you want to fully terminate the line in
the file.1 A subtler factor: the comma is a sort of lowest precedence operator, but only in contexts where it’s not otherwise significant. In such contexts, it’s the comma that builds tuples, not the parenthesis; this makes the latter optional, but can also lead to odd, unexpected syntax errors if parentheses are omitted.
| Statement | Role | Example |
|---|---|---|
a, b = 'good', 'bad' | ||
log.write("spam, ham") | ||
print('The Killer', joke) | ||
if "python" in text:
print(text) | ||
for x in mylist:
print(x) | ||
while X > Y:
print('hello') | ||
while True:
pass | ||
while True:
if exittest(): break | ||
while True:
if skiptest(): continue | ||
def f(a, b, c=1, *d):
print(a+b+c+d[0]) | ||
def f(a, b, c=1, *d):
return a+b+c+d[0] | ||
def gen(n):
for i in n: yield i*2 | ||
x = 'old'
def function():
global x, y; x = 'new' | ||
def outer():
x = 'old'
def function():
nonlocal x; x = 'new' | ||
import sys | ||
from sys import stdin | ||
class Subclass(Superclass):
staticData = []
def method(self): pass | ||
try:
action()
except:
print('action error') | ||
raise EndSearch(location) | ||
assert X > Y, 'X too small' | ||
with open('data') as myfile:
process(myfile) | ||
del data[k] del data[i:j] del obj.attr del variable |
print is technically neither
a reserved word nor a statement in 3.X, but a built-in function call;
because it will nearly always be run as an expression statement,
though (and often on a line by itself), it’s generally thought of as a
statement type. We’ll study print operations in Chapter 11.yield is also an expression
instead of a statement as of 2.5; like print, it’s typically used as an expression
statement and so is included in this table, but scripts occasionally
assign or otherwise use its result, as we’ll see in Chapter 20. As an expression,
yield is also a reserved word,
unlike print.nonlocal is not
available; as we’ll see in Chapter 17, there are
alternative ways to achieve this statement’s writeable state-retention
effect.print is a statement
instead of a built-in function call, with specific syntax covered in
Chapter 11.exec code
execution built-in function is a statement, with specific syntax;
since it supports enclosing parentheses, though, you can generally use
its 3.X call form in 2.X code.try/except and try/finally statements were merged: the two were
formerly separate statements, but we can now say both except and finally in the same try statement.with/as is an optional extension, and it is not
available unless you explicitly turn it on by running the statement from __future__
import with_statement (see Chapter 34).if (x > y) {
x = 1;
y = 2;
}if x > y:
x = 1
y = 2Header line:
Nested statement blockif (x < y)
if x < y
x = 1;
x = 1
if (x > y) {
x = 1;
y = 2;
}if x > y:
x = 1
y = 2while (x > 0) {while (x > 0) {
--------;
--------;while (x > 0) {
--------;
--------;
--------;
--------;while (x > 0) {
--------;
--------;
--------;
--------;
--------;
--------;
}if (x)
if (y)
statement1;
else
statement2;if x:
if y:
statement1
else:
statement2a = 1; b = 2; print(a + b) # Three statements on one linemylist = [1111,
2222,
3333]X = (A + B +
C + D)if (A == 1 and
B == 2 and
C == 3):
print('spam' * 3)X = A + B + \
C + D # An error-prone older alternativeif x > y: print(x)
while True:
reply = input('Enter text:')
if reply == 'stop': break
print(reply.upper())while loop,
Python’s most general looping statement. We’ll study the while statement in more detail later, but
in short, it consists of the word while, followed by an expression that is
interpreted as a true or false result, followed by a nested block of
code that is repeated while the test at the top is true (the word
True here is considered always
true).input built-in function
we met earlier in the book is used here for general
console input — it prints its optional argument string as a prompt and
returns the user’s typed reply as a string. Use raw_input in 2.X instead, per the upcoming
note.if statement
that makes use of the special rule for nested blocks also
appears here: the body of the if
appears on the header line after the colon instead of being indented
on a new line underneath it. This would work either way, but as it’s
coded, we’ve saved an extra line.break
statement is used to exit the loop immediately — it simply jumps out
of the loop statement altogether, and the program continues after
the loop. Without this exit statement, the while would loop forever, as its test is
always true.Enter text:spamSPAM Enter text:4242 Enter text:stop
import sys if sys.version[0] == '2': input = raw_input # 2.X compatible
>>>reply = '20'>>>reply ** 2...error text omitted...TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'
>>> int(reply) ** 2
400while True:
reply = input('Enter text:')
if reply == 'stop': break
print(int(reply) ** 2)
print('Bye')Enter text:24 Enter text:401600 Enter text:stopBye
Enter text:xxx...error text omitted...ValueError: invalid literal for int() with base 10: 'xxx'
>>>S = '123'>>>T = 'xxx'>>>S.isdigit(), T.isdigit()(True, False)
while True:
reply = input('Enter text:')
if reply == 'stop':
break
elif not reply.isdigit():
print('Bad!' * 8)
else:
print(int(reply) ** 2)
print('Bye')Enter text:525 Enter text:xyzBad!Bad!Bad!Bad!Bad!Bad!Bad!Bad! Enter text:10100 Enter text:stopBye
while True:
reply = input('Enter text:')
if reply == 'stop': break
try:
num = int(reply)
except:
print('Bad!' * 8)
else:
print(num ** 2)
print('Bye')while True:
reply = input('Enter text:')
if reply == 'stop': break
try:
print(int(reply) ** 2)
except:
print('Bad!' * 8)
print('Bye')while True:
reply = input('Enter text:')
if reply == 'stop': break
try:
print(float(reply) ** 2)
except:
print('Bad!' * 8)
print('Bye')Enter text:502500.0 Enter text:40.51640.25 Enter text:1.23E-1001.5129e-200 Enter text:spamBad!Bad!Bad!Bad!Bad!Bad!Bad!Bad! Enter text:stopBye
while True:
reply = input('Enter text:')
if reply == 'stop':
break
elif not reply.isdigit():
print('Bad!' * 8)
else:
num = int(reply)
if num < 20:
print('low')
else:
print(num ** 2)
print('Bye')Enter text:19low Enter text:20400 Enter text:spamBad!Bad!Bad!Bad!Bad!Bad!Bad!Bad! Enter text:stopBye
try statement
for?try statement is used to
catch and recover from exceptions (errors) in a Python script. It’s
usually an alternative to manually checking for errors in code.= statement, but assignment occurs
in many contexts in Python. For instance, we’ll see later that module
imports, function and class definitions, for loop variables, and function arguments
are all implicit assignments. Because assignment works the same
everywhere it pops up, all these contexts simply
bind (i.e., assign) names to object references at
runtime.| Operation | Interpretation |
|---|---|
The second and third forms in the table are related. When you code a tuple or list on the left side of the=, Python pairs objects on the right side with targets on the left by position and assigns them from left to right. For example, in the second line of Table 11-1, the namespamis assigned the string'yum', and the namehamis bound to the string'YUM'. In this case Python internally may make a tuple of the items on the right, which is why this is called tuple-unpacking assignment.
In later versions of Python, tuple and list assignments were generalized into instances of what we now call sequence assignment — any sequence of names can be assigned to any sequence of values, and Python assigns the items one at a time by position. We can even mix and match the types of the sequences involved. The fourth line in Table 11-1, for example, pairs a tuple of names with a string of characters:ais assigned's',bis assigned'p', and so on.
In Python 3.X (only), a new form of sequence assignment allows us to be more flexible in how we select portions of a sequence to assign. The fifth line in Table 11-1, for example, matchesawith the first character in the string on the right andbwith the rest:ais assigned's', andbis assigned['p', 'a', 'm']. This provides a simpler alternative to assigning the results of manual slicing operations.
The sixth line in Table 11-1 shows the multiple-target form of assignment. In this form, Python assigns a reference to the same object (the object farthest to the right) to all the targets on the left. In the table, the namesspamandhamare both assigned references to the same string object,'lunch'. The effect is the same as if we had codedham = 'lunch'followed byspam = ham, ashamevaluates to the original string object (i.e., not a separate copy of that object).
The last line in Table 11-1 is an example of augmented assignment — a shorthand that combines an expression and an assignment in a concise way. Sayingspam += 42, for example, has the same effect asspam = spam + 42, but the augmented form requires less typing and is generally quicker to run. In addition, if the subject is mutable and supports the operation, an augmented assignment may run even quicker by choosing an in-place update operation instead of an object copy. As we’ll see, there is one augmented assignment statement for most binary expression operators in Python.
%python>>>nudge = 1# Basic assignment>>>wink = 2>>>A, B = nudge, wink# Tuple assignment>>>A, B# Like A = nudge; B = wink(1, 2) >>>[C, D] = [nudge, wink]# List assignment>>>C, D(1, 2)
>>>nudge = 1>>>wink = 2>>>nudge, wink = wink, nudge# Tuples: swaps values>>>nudge, wink# Like T = nudge; nudge = wink; wink = T(2, 1)
>>>[a, b, c] = (1, 2, 3)# Assign tuple of values to list of names>>>a, c(1, 3) >>>(a, b, c) = "ABC"# Assign string of characters to tuple>>>a, c('A', 'C')
>>>string = 'SPAM'>>>a, b, c, d = string# Same number on both sides>>>a, d('S', 'M') >>>a, b, c = string# Error if not...error text omitted...ValueError: too many values to unpack (expected 3)
>>>a, b, c = string[0], string[1], string[2:]# Index and slice>>>a, b, c('S', 'P', 'AM') >>>a, b, c = list(string[:2]) + [string[2:]]# Slice and concatenate>>>a, b, c('S', 'P', 'AM') >>>a, b = string[:2]# Same, but simpler>>>c = string[2:]>>>a, b, c('S', 'P', 'AM') >>>(a, b), c = string[:2], string[2:]# Nested sequences>>>a, b, c('S', 'P', 'AM')
>>>((a, b), c) = ('SP', 'AM')# Paired by shape and position>>>a, b, c('S', 'P', 'AM')
for (a, b, c) in [(1, 2, 3), (4, 5, 6)]: ...# Simple tuple assignmentfor ((a, b), c) in [((1, 2), 3), ((4, 5), 6)]: ...# Nested tuple assignment
def f(((a, b), c)): ... # For arguments too in Python 2.X, but not 3.X
f(((1, 2), 3))>>>red, green, blue = range(3)>>>red, blue(0, 2)
>>>list(range(3))# list() required in Python 3.X only[0, 1, 2]
>>>L = [1, 2, 3, 4]>>>while L:...front, L = L[0], L[1:]# See next section for 3.X * alternative...print(front, L)... 1 [2, 3, 4] 2 [3, 4] 3 [4] 4 []
...front = L[0]...L = L[1:]
C:\code>c:\python33\python>>>seq = [1, 2, 3, 4]>>>a, b, c, d = seq>>>print(a, b, c, d)1 2 3 4 >>>a, b = seqValueError: too many values to unpack (expected 2)
>>>a, *b = seq>>>a1 >>>b[2, 3, 4]
>>>*a, b = seq>>>a[1, 2, 3] >>>b4
>>>a, *b, c = seq>>>a1 >>>b[2, 3] >>>c4
>>>a, b, *c = seq>>>a1 >>>b2 >>>c[3, 4]
>>>a, *b = 'spam'>>>a, b('s', ['p', 'a', 'm']) >>>a, *b, c = 'spam'>>>a, b, c('s', ['p', 'a'], 'm') >>>a, *b, c = range(4)>>>a, b, c(0, [1, 2], 3)
>>>S = 'spam'>>>S[0], S[1:]# Slices are type-specific, * assignment always returns a list('s', 'pam') >>>S[0], S[1:3], S[3]('s', 'pa', 'm')
>>>L = [1, 2, 3, 4]>>>while L:...front, *L = L# Get first, rest without slicing...print(front, L)... 1 [2, 3, 4] 2 [3, 4] 3 [4] 4 []
>>>seq = [1, 2, 3, 4]>>>a, b, c, *d = seq>>>print(a, b, c, d)1 2 3 [4]
>>>a, b, c, d, *e = seq>>>print(a, b, c, d, e)1 2 3 4 [] >>>a, b, *e, c, d = seq>>>print(a, b, c, d, e)1 2 3 4 []
>>>a, *b, c, *d = seqSyntaxError: two starred expressions in assignment >>>a, b = seqValueError: too many values to unpack (expected 2) >>>*a = seqSyntaxError: starred assignment target must be in a list or tuple >>>*a, = seq>>>a[1, 2, 3, 4]
>>>seq[1, 2, 3, 4] >>>a, *b = seq# First, rest>>>a, b(1, [2, 3, 4]) >>>a, b = seq[0], seq[1:]# First, rest: traditional>>>a, b(1, [2, 3, 4])
>>>*a, b = seq# Rest, last>>>a, b([1, 2, 3], 4) >>>a, b = seq[:-1], seq[-1]# Rest, last: traditional>>>a, b([1, 2, 3], 4)
for (a, *b, c) in [(1, 2, 3, 4), (5, 6, 7, 8)]:
...a, *b, c = (1, 2, 3, 4) # b gets [2, 3]for (a, b, c) in [(1, 2, 3), (4, 5, 6)]: # a, b, c = (1, 2, 3), ...for all in [(1, 2, 3, 4), (5, 6, 7, 8)]:
a, b, c = all[0], all[1:3], all[3]>>>a = b = c = 'spam'>>>a, b, c('spam', 'spam', 'spam')
>>>c = 'spam'>>>b = c>>>a = b
X = X + Y# Traditional formX += Y# Newer augmented form
>>>x = 1>>>x = x + 1# Traditional>>>x2 >>>x += 1# Augmented>>>x3
>>>S = "spam">>>S += "SPAM"# Implied concatenation>>>S'spamSPAM'
X += Y, X may be a complicated object expression.
In the augmented form, its code must be run only once. However, in
the long form, X = X + Y,
X appears twice and must be run
twice. Because of this, augmented assignments usually run
faster.>>>L = [1, 2]>>>L = L + [3]# Concatenate: slower>>>L[1, 2, 3] >>>L.append(4)# Faster, but in place>>>L[1, 2, 3, 4]
>>>L = L + [5, 6]# Concatenate: slower>>>L[1, 2, 3, 4, 5, 6] >>>L.extend([7, 8])# Faster, but in place>>>L[1, 2, 3, 4, 5, 6, 7, 8]
>>>L += [9, 10]# Mapped to L.extend([9, 10])>>>L[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>>L = []>>>L += 'spam'# += and extend allow any sequence, but + does not!>>>L['s', 'p', 'a', 'm'] >>>L = L + 'spam'TypeError: can only concatenate list (not "str") to list
Variable names must start with an underscore or letter, which can be followed by any number of letters, digits, or underscores._spam,spam, andSpam_1are legal names, but1_Spam,spam$, and@#!are not.
SPAM is not
the same as spamPython always pays attention to case in programs, both in names you create and in reserved words. For instance, the namesXandxrefer to two different variables. For portability, case also matters in the names of imported module files, even on platforms where the filesystems are case-insensitive. That way, your imports still work after programs are copied to differing platforms.
Names you define cannot be the same as words that mean special things in the Python language. For instance, if you try to use a variable name likeclass, Python will raise a syntax error, butklassandClasswork fine. Table 11-3 lists the words that are currently reserved (and hence off-limits for names of your own) in Python.
print is a reserved word,
because printing is a statement, not a built-in function (more on
this later in this chapter).exec is a reserved word,
because it is a statement, not a built-in function.nonlocal is not a reserved
word because this statement is not available.with and as were not reserved until 2.6, when
context managers were officially enabled.yield was not reserved
until Python 2.3, when generator functions came online.yield morphed from
statement to expression in 2.5, but it’s still a reserved word, not
a built-in function._X) are not imported by a from module import * statement
(described in Chapter 23).__X__) are system-defined
names that have special meaning to the interpreter.__X) are localized
(“mangled”) to enclosing classes (see the discussion of
pseudoprivate attributes in Chapter 31)._) retains the result of the last
expression when you are working interactively.>>>x = 0# x bound to an integer object>>>x = "Hello"# Now it's a string>>>x = [1, 2, 3]# And now it's a list
| Operation | Interpretation |
|---|---|
>>>x = print('spam')# print is a function call expression in 3.Xspam >>>print(x)# But it is coded as an expression statementNone
>>>L = [1, 2]>>>L.append(3)# Append is an in-place change>>>L[1, 2, 3]
>>>L = L.append(4)# But append returns None, not L>>>print(L)# So we lose our list!None
In Chapter 9, we learned about file object methods that write text (e.g.,file.write(str)). Printing operations are similar, but more focused — whereas file write methods write strings to arbitrary files,stdoutstream by default, with some automatic formatting added. Unlike with file methods, there is no need to convert objects to strings when using print operations.
The standard output stream (often known asstdout) is simply a default place to send a program’s text output. Along with the standard input and error streams, it’s one of three data connections created when your script starts. The standard output stream is usually mapped to the window where you started your Python program, unless it’s been redirected to a file or pipe in your operating system’s shell.
print([object, ...][, sep=' '][, end='\n'][, file=sys.stdout][, flush=False])
sep is a string inserted
between each object’s text, which defaults to a single space if
not passed; passing an empty string suppresses separators
altogether.end is a string added at
the end of the printed text, which defaults to a \n newline character if not passed.
Passing an empty string avoids dropping down to the next output
line at the end of the printed text — the next print will keep adding to the end of the
current output line.file specifies the file,
standard stream, or other file-like object to which the text will
be sent; it defaults to the sys.stdout standard output stream if not
passed. Any object with a file-like write(string) method may be passed, but real files
should be already opened for output.flush, added in 3.3,
defaults to False. It allows
prints to mandate that their text be flushed through the output
stream immediately to any waiting recipients. Normally, whether
printed output is buffered in memory or not is determined by
file; passing a true value to
flush forcibly flushes the
stream.C:\code>c:\python33\python>>>print()# Display a blank line>>>x = 'spam'>>>y = 99>>>z = ['eggs']>>> >>>print(x, y, z)# Print three objects per defaultsspam 99 ['eggs']
>>>print(x, y, z, sep='')# Suppress separatorspam99['eggs'] >>> >>>print(x, y, z, sep=', ')# Custom separatorspam, 99, ['eggs']
>>>print(x, y, z, end='')# Suppress line breakspam 99 ['eggs']>>> >>> >>>print(x, y, z, end=''); print(x, y, z)# Two prints, same output linespam 99 ['eggs']spam 99 ['eggs'] >>>print(x, y, z, end='...\n')# Custom line endspam 99 ['eggs']... >>>
>>>print(x, y, z, sep='...', end='!\n')# Multiple keywordsspam...99...['eggs']! >>>print(x, y, z, end='!\n', sep='...')# Order doesn't matterspam...99...['eggs']!
>>>print(x, y, z, sep='...', file=open('data.txt', 'w'))# Print to a file>>>print(x, y, z)# Back to stdoutspam 99 ['eggs'] >>>print(open('data.txt').read())# Display file textspam...99...['eggs']
>>>text = '%s: %-.4f, %05d' % ('Result', 3.14159, 42)>>>print(text)Result: 3.1416, 00042 >>>print('%s: %-.4f, %05d' % ('Result', 3.14159, 42))Result: 3.1416, 00042
| Python 2.X statement | Python 3.X equivalent | Interpretation |
|---|---|---|
C:\code>c:\python27\python>>>x = 'a'>>>y = 'b'>>>print x, ya b
>>> print x, y,; print x, y
a b a b>>>print x + yab >>>print '%s...%s' % (x, y)a...b
>>>print('hello world')# Print a string object in 3.Xhello world >>>print 'hello world'# Print a string object in 2.Xhello world
>>>'hello world'# Interactive echoes'hello world'
>>>import sys# Printing the hard way>>>sys.stdout.write('hello world\n')hello world
print(X, Y) # Or, in 2.X: print X, Yimport sys sys.stdout.write(str(X) + ' ' + str(Y) + '\n')
import sys
sys.stdout = open('log.txt', 'a') # Redirects prints to a file
...
print(x, y, x) # Shows up in log.txtC:\code>c:\python33\python>>>import sys>>>temp = sys.stdout# Save for restoring later>>>sys.stdout = open('log.txt', 'a')# Redirect prints to a file>>>print('spam')# Prints go to file, not here>>>print(1, 2, 3)>>>sys.stdout.close()# Flush output to disk>>>sys.stdout = temp# Restore original stream>>>print('back here')# Prints show up here againback here >>>print(open('log.txt').read())# Result of earlier printsspam 1 2 3
log = open('log.txt', 'a') # 3.X
print(x, y, z, file=log) # Print to a file-like object
print(a, b, c) # Print to original stdout
log = open('log.txt', 'a') # 2.X
print >> log, x, y, z # Print to a file-like object
print a, b, c # Print to original stdoutC:\code>c:\python33\python>>>log = open('log.txt', 'w')>>>print(1, 2, 3, file=log)# For 2.X: print >> log, 1, 2, 3>>>print(4, 5, 6, file=log)>>>log.close()>>>print(7, 8, 9)# For 2.X: print 7, 8, 97 8 9 >>>print(open('log.txt').read())1 2 3 4 5 6
>>>import sys>>>sys.stderr.write(('Bad!' * 8) + '\n')Bad!Bad!Bad!Bad!Bad!Bad!Bad!Bad! >>>print('Bad!' * 8, file=sys.stderr)# In 2.X: print >> sys.stderr, 'Bad!' * 8Bad!Bad!Bad!Bad!Bad!Bad!Bad!Bad!
>>>X = 1; Y = 2>>>print(X, Y)# Print: the easy way1 2 >>>import sys# Print: the hard way>>>sys.stdout.write(str(X) + ' ' + str(Y) + '\n')1 2 4 >>>print(X, Y, file=open('temp1', 'w'))# Redirect text to file>>>open('temp2', 'w').write(str(X) + ' ' + str(Y) + '\n')# Send to file manually4 >>>print(open('temp1', 'rb').read())# Binary mode for bytesb'1 2\r\n' >>>print(open('temp2', 'rb').read())b'1 2\r\n'
from __future__ import print_function
C:\code>c:\python33\python>>>print('spam')# 3.X print function call syntaxspam >>>print('spam', 'ham', 'eggs')# These are multiple argumentsspam ham eggs
C:\code>c:\python27\python>>>print('spam')# 2.X print statement, enclosing parensspam >>>print('spam', 'ham', 'eggs')# This is really a tuple object!('spam', 'ham', 'eggs')
c:\code>py −2>>print()# This is just a line-feed on 3.X() >>>print('')# This is a line-feed in both 2.X and 3.X
>>>print('%s %s %s' % ('spam', 'ham', 'eggs'))spam ham eggs >>>print('{0} {1} {2}'.format('spam', 'ham', 'eggs'))spam ham eggs >>>print('answer: ' + str(42))answer: 42
A = B = C = 0), sequence assignment
(A, B, C = 0, 0, 0), or multiple
assignment statements on three separate lines (A = 0, B =
0, and C = 0). With the
latter technique, as introduced in Chapter 10, you can also string the
three separate statements together on the same line by separating them
with semicolons (A = 0; B = 0; C =
0).A = B = C = []
A.append(99))
will affect the others. This is true only for in-place changes to
mutable objects like lists and dictionaries; for immutable objects
such as numbers and strings, this issue is irrelevant.sort method is like
append in that it makes an in-place
change to the subject list — it returns None, not the list it changes. The
assignment back to L sets L to None, not to the sorted list. As discussed
both earlier and later in this book (e.g., Chapter 8), a newer built-in function,
sorted, sorts any sequence and
returns a new list with the sorting result; because this is not an
in-place change, its result can be meaningfully assigned to a
name.print operation, you can use 3.X’s print(X, file=F) call form, use 2.X’s
extended print >> file, X
statement form, or assign sys.stdout to a manually opened file before
the print and restore the original
after. You can also redirect all of a program’s printed text to a file
with special syntax in the system shell, but this is outside Python’s
scope.1 C/C++ programmers take note: although Python now supports
statements like X += Y, it still
does not have C’s auto-increment/decrement operators (e.g., X++, −−X). These don’t quite map to the Python
object model because Python has no notion of
in-place changes to immutable objects like
numbers.
2 As suggested in Chapter 6, we can also use slice
assignment (e.g., L[len(L):] =
[11,12,13]), but this works roughly the same as the
simpler and more mnemonic list extend method.
3 In standard CPython, at least. Alternative implementations of Python might allow user-defined variable names to be the same as Python reserved words. See Chapter 2 for an overview of alternative implementations, such as Jython.
4 If you’ve used a more restrictive language like C++, you may
be interested to know that there is no notion of C++’s const declaration in Python; certain
objects may be immutable, but names can
always be assigned. Python also has ways to hide names in classes
and modules, but they’re not the same as C++’s declarations (if
hiding attributes matters to you, see the coverage of _X module names in Chapter 25, __X class names in Chapter 31, and the Private and Public
class decorators example in Chapter 39).
5 Technically, printing uses the equivalent of str in the internal implementation of
Python, but the effect is the same. Besides this to-string
conversion role, str is also
the name of the string data type and can be used to decode Unicode
strings from raw bytes with an extra encoding argument, as we’ll
learn in Chapter 37; this latter
role is an advanced usage that we can safely ignore here.
6 In both 2.X and 3.X you may also be able to use the __stdout__ attribute in the sys module, which refers to the original
value sys.stdout had at program
startup time. You still need to restore sys.stdout to sys.__stdout__ to go back to this
original stream value, though. See the sys module documentation for more
details.
iftest1:# if teststatements1# Associated blockeliftest2:# Optional elifsstatements2else:# Optional elsestatements3
>>>if 1:...print('true')... true
>>>if not 1:...print('true')...else:...print('false')... false
>>>x = 'killer rabbit'>>>if x == 'roger':...print("shave and a haircut")...elif x == 'bugs':...print("what's up doc?")...else:...print('Run away! Run away!')... Run away! Run away!
>>>choice = 'ham'>>>print({'spam': 1.25,# A dictionary-based 'switch'...'ham': 1.99,# Use has_key or get for default...'eggs': 0.99,...'bacon': 1.10}[choice])1.99
>>>if choice == 'spam':# The equivalent if statement...print(1.25)...elif choice == 'ham':...print(1.99)...elif choice == 'eggs':...print(0.99)...elif choice == 'bacon':...print(1.10)...else:...print('Bad choice')... 1.99
>>>branch = {'spam': 1.25,...'ham': 1.99,...'eggs': 0.99}>>>print(branch.get('spam', 'Bad choice'))1.25 >>>print(branch.get('bacon', 'Bad choice'))Bad choice
>>>choice = 'bacon'>>>if choice in branch:...print(branch[choice])...else:...print('Bad choice')... Bad choice
>>>try:...print(branch[choice])...except KeyError:...print('Bad choice')... Bad choice
def function(): ...def default(): ...branch = {'spam': lambda: ...,# A table of callable function objects'ham': function,'eggs': lambda: ...}branch.get(choice, default)()
if
(as well as loops and exceptions) cause the interpreter to jump around
in your code. Because Python’s path through a program is called the
control flow, statements such as if that affect it are often called
control-flow statements.if
statement, the elif and else clauses are part of the if, but they are also header lines with
nested blocks of their own. As a special case, blocks can show up on
the same line as the header if they are simple noncompound
code.# character (not inside a
string literal) and extend to the end of the current line.#
comments, are retained at runtime for inspection. Docstrings are
simply strings that show up at the top of program files and some
statements. Python ignores their contents, but they are automatically
attached to objects at runtime and may be displayed with documentation
tools like PyDoc. Docstrings are part of Python’s larger documentation
strategy and are covered in the last chapter in this part of the
book.x = 1
if x:
y = 2
if y:
print('block2')
print('block1')
print('block0')x = 'SPAM'# Error: first line indentedif 'rubbery' in 'shrubbery': print(x * 8) x += 'NI'# Error: unexpected indentationif x.endswith('NI'): x *= 2 print(x)# Error: inconsistent indentation
x = 'SPAM'
if 'rubbery' in 'shrubbery':
print(x * 8) # Prints 8 "SPAM"
x += 'NI'
if x.endswith('NI'):
x *= 2
print(x) # Prints "SPAMNISPAMNI"(),
{}, or [] pair. For instance, expressions in
parentheses and dictionary and list literals can span any number of
lines; your statement doesn’t end until the Python interpreter
reaches the line on which you type the closing part of the pair (a
), }, or ]). Continuation
lines — lines 2 and beyond of the statement — can start at any
indentation level you like, but you should try to make them align
vertically for readability if possible. This open pairs rule also
covers set and dictionary comprehensions in Python 3.X and
2.7.\ not
embedded in a string literal or comment) at the end of the prior
line to indicate you’re continuing on the next line. Because you can
also continue by adding parentheses around most constructs,
backslashes are rarely used today. This approach is also
error-prone: accidentally forgetting a \ usually generates a syntax error and
might even cause the next line to be silently mistaken (i.e.,
without warning) for a new statement, with unexpected
results.# character) terminate at
the end of the line on which they appear.L = ["Good",
"Bad",
"Ugly"] # Open pairs may span linesif a == b and c == d and \
d == e and f == g:
print('olde') # Backslashes allow continuations...if (a == b and c == d and
d == e and e == f):
print('new') # But parentheses usually do too, and are obviousx = 1 + 2 + 3 \ # Omitting the \ makes this very different!
+4x = 1; y = 2; print(x) # More than one simple statementS = """
aaaa
bbbb
cccc"""
S = ('aaaa'
'bbbb' # Comments here are ignored
'cccc')if 1: print('hello') # Simple statement on header lineNone are considered false.True or False (custom versions of 1 and 0).and and or operators return a true or false operand
object.X and YIs true if bothXandYare true
X or YIs true if eitherXorYis true
not XIs true ifXis false (the expression returnsTrueorFalse)
>>>2 < 3, 3 < 2# Less than: return True or False (1 or 0)(True, False)
>>>2 or 3, 3 or 2# Return left operand if true(2, 3)# Else, return right operand (true or false)>>>[] or 33 >>>[] or {}{}
>>>2 and 3, 3 and 2# Return left operand if false(3, 2)# Else, return right operand (true or false)>>>[] and {}[] >>>3 and [][]
if X:
A = Y
else:
A = ZA = Y if X else Z
>>>A = 't' if 'spam' else 'f'# For strings, nonempty means true>>>A't' >>>A = 't' if '' else 'f'>>>A'f'
A = ((X and Y) or Z)
A = Y if X else Z
A = [Z, Y][bool(X)]
>>>['f', 't'][bool('')]'f' >>>['f', 't'][bool('spam')]'t'
if statement with multiple
elif clauses is often the most
straightforward way to code a multiway branch, though not necessarily
the most concise or flexible. Dictionary indexing can often achieve
the same result, especially if the dictionary contains callable
functions coded with def statements
or lambda expressions.Y if X else Z returns Y if X is
true, or Z otherwise; it’s the same
as a four-line if statement. The
and/or combination (((X
and Y) or Z)) can work the same way, but it’s more obscure
and requires that the Y part be
true.(), [],
or {}), and it can span as many
lines as you like; the statement ends when Python sees the closing
(right) half of the pair, and lines 2 and beyond of the statement can
begin at any indentation level. Backslash continuations work too, but
are broadly discouraged in the Python world.True and False are just custom versions of the
integers 1 and 0, respectively: they always stand for
Boolean true and false values in Python. They’re available for use in
truth tests and variable initialization, and are printed for
expression results at the interactive prompt. In all these roles, they
serve as a more mnemonic and hence readable alternative to 1 and 0.1 Candidly, it was a bit surprising that backslash continuations were not removed in Python 3.0, given the broad scope of its other changes! See the 3.0 changes tables in Appendix C for a list of 3.0 removals; some seem fairly innocuous in comparison with the dangers inherent in backslash continuations. Then again, this book’s goal is Python instruction, not populist outrage, so the best advice I can give is simply: don’t do this. You should generally avoid backslash continuations in new Python code, even if you developed the habit in your C programming days.
2 In fact, Python’s Y if X else
Z has a slightly different order than C’s X ? Y : Z, and uses more readable words. Its
differing order was reportedly chosen in response to analysis of
common usage patterns in Python code. According to the Python
folklore, this order was also chosen in part to discourage ex–C
programmers from overusing it! Remember, simple is better than
complex, in Python and elsewhere. If you have to work at packing logic
into expressions like this, statements are probably your better
bet.
whiletest:# Loop teststatements# Loop bodyelse:# Optional elsestatements# Run if didn't exit loop with break
>>>while True:...print('Type Ctrl-C to stop me!')
>>>x = 'spam'>>>while x:# While x is not empty...print(x, end=' ')# In 2.X use print x,...x = x[1:]# Strip first character off x... spam pam am m
>>>a=0; b=10>>>while a < b:# One way to code counter loops...print(a, end=' ')...a += 1# Or, a = a + 1... 0 1 2 3 4 5 6 7 8 9
while True:
...loop body...
if exitTest(): breakbreakcontinuepassLoop else blockwhiletest:statementsiftest: break# Exit loop now, skip else if presentiftest: continue# Go to test at top of loop nowelse:statements# Run if we didn't hit a 'break'
while True: pass # Type Ctrl-C to stop me!def func1():
pass # Add real code here later
def func2():
passdef func1():
... # Alternative to pass
def func2():
...
func1() # Does nothing if calleddef func1(): ...# Works on same line toodef func2(): ... >>> X = ...# Alternative to None>>> X Ellipsis
x = 10
while x:
x = x−1 # Or, x -= 1
if x % 2 != 0: continue # Odd? -- skip print
print(x, end=' ')x = 10
while x:
x = x−1
if x % 2 == 0: # Even? -- print
print(x, end=' ')>>>while True:...name = input('Enter name:')# Use raw_input() in 2.X...if name == 'stop': break...age = input('Enter age: ')...print('Hello', name, '=>', int(age) ** 2)... Enter name:bobEnter age:40Hello bob => 1600 Enter name:sueEnter age:30Hello sue => 900 Enter name:stop
x = y // 2# For some y > 1while x > 1: if y % x == 0:# Remainderprint(y, 'has factor', x) break# Skip elsex -= 1 else:# Normal exitprint(y, 'is prime')
found = False
while x and not found:
if match(x[0]): # Value at front?
print('Ni')
found = True
else:
x = x[1:] # Slice off front and repeat
if not found:
print('not found')while x:# Exit when x emptyif match(x[0]): print('Ni') break# Exit, go around elsex = x[1:] else: print('Not found')# Only here if exhausted x
fortargetinobject:# Assign object items to targetstatements# Repeated loop body: use targetelse:# Optional else partstatements# If we didn't hit a 'break'
fortargetinobject:# Assign object items to targetstatementsiftest: break# Exit loop now, skip elseiftest: continue# Go to top of loop nowelse:statements# If we didn't hit a 'break'
>>>for x in ["spam", "eggs", "ham"]:...print(x, end=' ')... spam eggs ham
>>>sum = 0>>>for x in [1, 2, 3, 4]:...sum = sum + x... >>>sum10 >>>prod = 1>>>for item in [1, 2, 3, 4]: prod *= item... >>>prod24
>>>S = "lumberjack">>>T = ("and", "I'm", "okay")>>>for x in S: print(x, end=' ')# Iterate over a string... l u m b e r j a c k >>>for x in T: print(x, end=' ')# Iterate over a tuple... and I'm okay
>>>T = [(1, 2), (3, 4), (5, 6)]>>>for (a, b) in T:# Tuple assignment at work...print(a, b)... 1 2 3 4 5 6
>>>D = {'a': 1, 'b': 2, 'c': 3}>>>for key in D:...print(key, '=>', D[key])# Use dict keys iterator and index... a => 1 c => 3 b => 2 >>>list(D.items())[('a', 1), ('c', 3), ('b', 2)] >>>for (key, value) in D.items():...print(key, '=>', value)# Iterate over both keys and values... a => 1 c => 3 b => 2
>>>T[(1, 2), (3, 4), (5, 6)] >>>for both in T:...a, b = both# Manual assignment equivalent...print(a, b)# 2.X: prints with enclosing tuple "()"... 1 2 3 4 5 6
>>>((a, b), c) = ((1, 2), 3)# Nested sequences work too>>>a, b, c(1, 2, 3) >>>for ((a, b), c) in [((1, 2), 3), ((4, 5), 6)]: print(a, b, c)... 1 2 3 4 5 6
>>> for ((a, b), c) in [([1, 2], 3), ['XY', 6]]: print(a, b, c)
...
1 2 3
X Y 6>>>a, b, c = (1, 2, 3)# Tuple assignment>>>a, b, c(1, 2, 3) >>>for (a, b, c) in [(1, 2, 3), (4, 5, 6)]:# Used in for loop...print(a, b, c)... 1 2 3 4 5 6
>>>a, *b, c = (1, 2, 3, 4)# Extended seq assignment>>>a, b, c(1, [2, 3], 4) >>>for (a, *b, c) in [(1, 2, 3, 4), (5, 6, 7, 8)]:...print(a, b, c)... 1 [2, 3] 4 5 [6, 7] 8
>>>for all in [(1, 2, 3, 4), (5, 6, 7, 8)]:# Manual slicing in 2.X...a, b, c = all[0], all[1:3], all[3]...print(a, b, c)... 1 (2, 3) 4 5 (6, 7) 8
>>>items = ["aaa", 111, (4, 5), 2.01]# A set of objects>>>tests = [(4, 5), 3.14]# Keys to search for>>> >>>for key in tests:# For all keys...for item in items:# For all items...if item == key:# Check for match...print(key, "was found")...break...else:...print(key, "not found!")... (4, 5) was found 3.14 not found!
>>>for key in tests:# For all keys...if key in items:# Let Python check for a match...print(key, "was found")...else:...print(key, "not found!")... (4, 5) was found 3.14 not found!
>>>seq1 = "spam">>>seq2 = "scam">>> >>>res = []# Start empty>>>for x in seq1:# Scan first sequence...if x in seq2:# Common item?...res.append(x)# Add to result end... >>>res['s', 'a', 'm']
>>>[x for x in seq1 if x in seq2]# Let Python collect results['s', 'a', 'm']
range function
(available since Python 0.X) produces a series of
successively higher integers, which can be used as indexes in a
for.zip function
(available since Python 2.0) returns a series of
parallel-item tuples, which can be used to traverse multiple sequences
in a for.enumerate
function (available since Python 2.3) generates both the values
and indexes of items in an iterable, so we don’t need to count
manually.map function
(available since Python 1.0) can have a similar effect to
zip in Python 2.X, though this role
is removed in 3.X.>>> list(range(5)), list(range(2, 5)), list(range(0, 10, 2))
([0, 1, 2, 3, 4], [2, 3, 4], [0, 2, 4, 6, 8])>>>list(range(−5, 5))[−5, −4, −3, −2, −1, 0, 1, 2, 3, 4] >>>list(range(5, −5, −1))[5, 4, 3, 2, 1, 0, −1, −2, −3, −4]
>>>for i in range(3):...print(i, 'Pythons')... 0 Pythons 1 Pythons 2 Pythons
>>>X = 'spam'>>>for item in X: print(item, end=' ')# Simple iteration... s p a m
>>>i = 0>>>while i < len(X):# while loop iteration...print(X[i], end=' ')...i += 1... s p a m
>>>X'spam' >>>len(X)# Length of string4 >>>list(range(len(X)))# All legal offsets into X[0, 1, 2, 3] >>> >>>for i in range(len(X)): print(X[i], end=' ')# Manual range/len iteration... s p a m
>>>for item in X: print(item, end=' ')# Use simple iteration if you can
>>>S = 'spam'>>>for i in range(len(S)):# For repeat counts 0..3...S = S[1:] + S[:1]# Move front item to end...print(S, end=' ')... pams amsp mspa spam >>>S'spam' >>>for i in range(len(S)):# For positions 0..3...X = S[i:] + S[:i]# Rear part + front part...print(X, end=' ')... spam pams amsp mspa
>>>L = [1, 2, 3]>>>for i in range(len(L)):...X = L[i:] + L[:i]# Works on any sequence type...print(X, end=' ')... [1, 2, 3] [2, 3, 1] [3, 1, 2]
>>>S = 'abcdefghijk'>>>list(range(0, len(S), 2))[0, 2, 4, 6, 8, 10] >>>for i in range(0, len(S), 2): print(S[i], end=' ')... a c e g i k
>>>S = 'abcdefghijk'>>>for c in S[::2]:print(c, end=' ')... a c e g i k
>>>L = [1, 2, 3, 4, 5]>>>for x in L:...x += 1# Changes x, not L... >>>L[1, 2, 3, 4, 5] >>>x6
>>>L = [1, 2, 3, 4, 5]>>>for i in range(len(L)):# Add one to each item in L...L[i] += 1# Or L[i] = L[i] + 1... >>>L[2, 3, 4, 5, 6]
>>>i = 0>>>while i < len(L):...L[i] += 1...i += 1... >>>L[3, 4, 5, 6, 7]
[x + 1 for x in L]
>>>L1 = [1,2,3,4]>>>L2 = [5,6,7,8]
>>>zip(L1, L2)<zip object at 0x026523C8> >>>list(zip(L1, L2))# list() required in 3.X, not 2.X[(1, 5), (2, 6), (3, 7), (4, 8)]
>>>for (x, y) in zip(L1, L2):...print(x, y, '--', x+y)... 1 5 -- 6 2 6 -- 8 3 7 -- 10 4 8 -- 12
>>>T1, T2, T3 = (1,2,3), (4,5,6), (7,8,9)>>>T3(7, 8, 9) >>>list(zip(T1, T2, T3))# Three tuples for three arguments[(1, 4, 7), (2, 5, 8), (3, 6, 9)]
>>>S1 = 'abc'>>>S2 = 'xyz123'>>> >>>list(zip(S1, S2))# Truncates at len(shortest)[('a', 'x'), ('b', 'y'), ('c', 'z')]
>>>S1 = 'abc'>>>S2 = 'xyz123'>>>map(None, S1, S2)# 2.X only: pads to len(longest)[('a', 'x'), ('b', 'y'), ('c', 'z'), (None, '1'), (None, '2'), (None,'3')]
>>> list(map(ord, 'spam'))
[115, 112, 97, 109]>>>res = []>>>for c in 'spam': res.append(ord(c))>>>res[115, 112, 97, 109]
>>>D1 = {'spam':1, 'eggs':3, 'toast':5}>>>D1{'eggs': 3, 'toast': 5, 'spam': 1} >>>D1 = {}>>>D1['spam'] = 1>>>D1['eggs'] = 3>>>D1['toast'] = 5
>>>keys = ['spam', 'eggs', 'toast']>>>vals = [1, 3, 5]
>>>list(zip(keys, vals))[('spam', 1), ('eggs', 3), ('toast', 5)] >>>D2 = {}>>>for (k, v) in zip(keys, vals): D2[k] = v... >>>D2{'eggs': 3, 'toast': 5, 'spam': 1}
>>>keys = ['spam', 'eggs', 'toast']>>>vals = [1, 3, 5]>>>D3 = dict(zip(keys, vals))>>>D3{'eggs': 3, 'toast': 5, 'spam': 1}
>>> {k: v for (k, v) in zip(keys, vals)}
{'eggs': 3, 'toast': 5, 'spam': 1}>>>S = 'spam'>>>offset = 0>>>for item in S:...print(item, 'appears at offset', offset)...offset += 1... s appears at offset 0 p appears at offset 1 a appears at offset 2 m appears at offset 3
>>>S = 'spam'>>>for (offset, item) in enumerate(S):...print(item, 'appears at offset', offset)... s appears at offset 0 p appears at offset 1 a appears at offset 2 m appears at offset 3
>>>E = enumerate(S)>>>E<enumerate object at 0x0000000002A8B900> >>>next(E)(0, 's') >>>next(E)(1, 'p') >>>next(E)(2, 'a')
>>>[c * i for (i, c) in enumerate(S)]['', 'p', 'aa', 'mmm'] >>>for (i, l) in enumerate(open('test.txt')):... print('%s) %s' % (i, l.rstrip()))... 0) aaaaaa 1) bbbbbb 2) cccccc
while loop is a general
looping statement, but the for is
designed to iterate across items in a sequence or other iterable.
Although the while can imitate the
for with counter loops, it takes
more code and might run slower.break statement exits a
loop immediately (you wind up below the entire while or for loop statement), and continue jumps back to the top of the loop
(you wind up positioned just before the test in while or the next item fetch in for).else clause in a while or for loop will be run once as the loop is
exiting, if the loop exits normally (without running into a break statement). A break exits the loop immediately, skipping
the else part on the way out (if
there is one).while statement that keeps track of the
index manually, or with a for loop
that uses the range built-in
function to generate successive integer offsets. Neither is the
preferred way to work in Python, if you need to simply step across all
the items in a sequence. Instead, use a simple for loop instead, without range or counters, whenever possible; it
will be easier to code and usually quicker to run.range built-in can be
used in a for to implement a fixed
number of repetitions, to scan by offsets instead of items at offsets,
to skip successive items as you go, and to change a list while
stepping across it. None of these roles requires range, and most have alternatives — scanning
actual items, three-limit slices, and list comprehensions are often
better solutions today (despite the natural inclinations of ex–C
programmers to want to count things!).>>>for x in [1, 2, 3, 4]: print(x ** 2, end=' ')# In 2.X: print x ** 2,... 1 4 9 16 >>>for x in (1, 2, 3, 4): print(x ** 3, end=' ')... 1 8 27 64 >>>for x in 'spam': print(x * 2, end=' ')... ss pp aa mm
>>>print(open('script2.py').read())import sys print(sys.path) x = 2 print(x ** 32) >>>open('script2.py').read()'import sys\nprint(sys.path)\nx = 2\nprint(x ** 32)\n'
>>>f = open('script2.py')# Read a four-line script file in this directory>>>f.readline()# readline loads one line on each call'import sys\n' >>>f.readline()'print(sys.path)\n' >>>f.readline()'x = 2\n' >>>f.readline()# Last lines may have a \n or not'print(x ** 32)\n' >>>f.readline()# Returns empty string at end-of-file''
>>>f = open('script2.py')# __next__ loads one line on each call too>>>f.__next__()# But raises an exception at end-of-file'import sys\n' >>>f.__next__()# Use f.next() in 2.X, or next(f) in 2.X or 3.X'print(sys.path)\n' >>>f.__next__()'x = 2\n' >>>f.__next__()'print(x ** 32)\n' >>>f.__next__()Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
>>>for line in open('script2.py'):# Use file iterators to read by lines...print(line.upper(), end='')# Calls __next__, catches StopIteration... IMPORT SYS PRINT(SYS.PATH) X = 2 PRINT(X ** 32)
>>>for line in open('script2.py').readlines():...print(line.upper(), end='')... IMPORT SYS PRINT(SYS.PATH) X = 2 PRINT(X ** 32)
>>>f = open('script2.py')>>>while True:...line = f.readline()...if not line: break...print(line.upper(), end='')... ...same output...
>>>f = open('script2.py')>>>f.__next__()# Call iteration method directly'import sys\n' >>>f.__next__()'print(sys.path)\n' >>>f = open('script2.py')>>>next(f)# The next(f) built-in calls f.__next__() in 3.X'import sys\n' >>>next(f)# next(f) => [3.X: f.__next__()], [2.X: f.next()]'print(sys.path)\n'
__iter__
is run by iter__next__ is run by next and raises StopIteration when finished producing results>>>L = [1, 2, 3]>>>I = iter(L)# Obtain an iterator object from an iterable>>>I.__next__()# Call iterator's next to advance to next item1 >>>I.__next__()# Or use I.next() in 2.X, next(I) in either line2 >>>I.__next__()3 >>>I.__next__()...error text omitted...StopIteration
>>>f = open('script2.py')>>>iter(f) is fTrue >>>iter(f) is f.__iter__()True >>>f.__next__()'import sys\n'
>>>L = [1, 2, 3]>>>iter(L) is LFalse >>>L.__next__()AttributeError: 'list' object has no attribute '__next__' >>>I = iter(L)>>>I.__next__()1 >>>next(I)# Same as I.__next__()2
>>>L = [1, 2, 3]>>> >>>for X in L:# Automatic iteration...print(X ** 2, end=' ')# Obtains iter, calls __next__, catches exceptions... 1 4 9 >>>I = iter(L)# Manual iteration: what for loops usually do>>>while True:...try:# try statement catches exceptions...X = next(I)# Or call I.__next__ in 3.X...except StopIteration:...break...print(X ** 2, end=' ')... 1 4 9
>>>D = {'a':1, 'b':2, 'c':3}>>>for key in D.keys():...print(key, D[key])... a 1 b 2 c 3
>>>I = iter(D)>>>next(I)'a' >>>next(I)'b' >>>next(I)'c' >>>next(I)Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
>>>for key in D:...print(key, D[key])... a 1 b 2 c 3
>>>import os>>>P = os.popen('dir')>>>P.__next__()' Volume in drive C has no label.\n' >>>P.__next__()' Volume Serial Number is D093-D1F7\n' >>>next(P)TypeError: _wrap_close object is not an iterator
>>>P = os.popen('dir')>>>I = iter(P)>>>next(I)' Volume in drive C has no label.\n' >>>I.__next__()' Volume Serial Number is D093-D1F7\n'
>>>R = range(5)>>>R# Ranges are iterables in 3.Xrange(0, 5) >>>I = iter(R)# Use iteration protocol to produce results>>>next(I)0 >>>next(I)1 >>>list(range(5))# Or use list to collect all results at once[0, 1, 2, 3, 4]
>>>E = enumerate('spam')# enumerate is an iterable too>>>E<enumerate object at 0x00000000029B7678> >>>I = iter(E)>>>next(I)# Generate results with iteration protocol(0, 's') >>>next(I)# Or use list to force generation to run(1, 'p') >>>list(enumerate('spam'))[(0, 's'), (1, 'p'), (2, 'a'), (3, 'm')]
>>>L = [1, 2, 3, 4, 5]>>>for i in range(len(L)):...L[i] += 10... >>>L[11, 12, 13, 14, 15]
>>>L = [x + 10 for x in L]>>>L[21, 22, 23, 24, 25]
L = [x + 10 for x in L]
>>>res = []>>>for x in L:...res.append(x + 10)... >>>res[31, 32, 33, 34, 35]
>>>f = open('script2.py')>>>lines = f.readlines()>>>lines['import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n']
>>>lines = [line.rstrip() for line in lines]>>>lines['import sys', 'print(sys.path)', 'x = 2', 'print(x ** 32)']
>>>lines = [line.rstrip() for line in open('script2.py')]>>>lines['import sys', 'print(sys.path)', 'x = 2', 'print(x ** 32)']
>>>[line.upper() for line in open('script2.py')]['IMPORT SYS\n', 'PRINT(SYS.PATH)\n', 'X = 2\n', 'PRINT(X ** 32)\n'] >>>[line.rstrip().upper() for line in open('script2.py')]['IMPORT SYS', 'PRINT(SYS.PATH)', 'X = 2', 'PRINT(X ** 32)'] >>>[line.split() for line in open('script2.py')][['import', 'sys'], ['print(sys.path)'], ['x', '=', '2'], ['print(x', '**', '32)']] >>>[line.replace(' ', '!') for line in open('script2.py')]['import!sys\n', 'print(sys.path)\n', 'x!=!2\n', 'print(x!**!32)\n'] >>>[('sys' in line, line[:5]) for line in open('script2.py')][(True, 'impor'), (True, 'print'), (False, 'x = 2'), (False, 'print')]
>>>lines = [line.rstrip() for line in open('script2.py') if line[0] == 'p']>>>lines['print(sys.path)', 'print(x ** 32)']
>>>res = []>>>for line in open('script2.py'):...if line[0] == 'p':...res.append(line.rstrip())... >>>res['print(sys.path)', 'print(x ** 32)']
>>> [line.rstrip() for line in open('script2.py') if line.rstrip()[-1].isdigit()]
['x = 2']>>>fname = r'd:\books\5e\lp5e\draft1typos.txt'>>>len(open(fname).readlines())# All lines263 >>>len([line for line in open(fname) if line.strip() != ''])# Nonblank lines185
>>> [x + y for x in 'abc' for y in 'lmn']
['al', 'am', 'an', 'bl', 'bm', 'bn', 'cl', 'cm', 'cn']>>>res = []>>>for x in 'abc':...for y in 'lmn':...res.append(x + y)... >>>res['al', 'am', 'an', 'bl', 'bm', 'bn', 'cl', 'cm', 'cn']
>>>for line in open('script2.py'):# Use file iterators...print(line.upper(), end='')... IMPORT SYS PRINT(SYS.PATH) X = 2 PRINT(X ** 32)
>>>uppers = [line.upper() for line in open('script2.py')]>>>uppers['IMPORT SYS\n', 'PRINT(SYS.PATH)\n', 'X = 2\n', 'PRINT(X ** 32)\n'] >>>map(str.upper, open('script2.py'))# map is itself an iterable in 3.X<map object at 0x00000000029476D8> >>>list(map(str.upper, open('script2.py')))['IMPORT SYS\n', 'PRINT(SYS.PATH)\n', 'X = 2\n', 'PRINT(X ** 32)\n']
>>>sorted(open('script2.py'))['import sys\n', 'print(sys.path)\n', 'print(x ** 32)\n', 'x = 2\n'] >>>list(zip(open('script2.py'), open('script2.py')))[('import sys\n', 'import sys\n'), ('print(sys.path)\n', 'print(sys.path)\n'), ('x = 2\n', 'x = 2\n'), ('print(x ** 32)\n', 'print(x ** 32)\n')] >>>list(enumerate(open('script2.py')))[(0, 'import sys\n'), (1, 'print(sys.path)\n'), (2, 'x = 2\n'), (3, 'print(x ** 32)\n')] >>>list(filter(bool, open('script2.py')))# nonempty=True['import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n'] >>>import functools, operator>>>functools.reduce(operator.add, open('script2.py'))'import sys\nprint(sys.path)\nx = 2\nprint(x ** 32)\n'
>>>list(open('script2.py'))['import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n'] >>>tuple(open('script2.py'))('import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n') >>>'&&'.join(open('script2.py'))'import sys\n&&print(sys.path)\n&&x = 2\n&&print(x ** 32)\n'
>>>a, b, c, d = open('script2.py')# Sequence assignment>>>a, d('import sys\n', 'print(x ** 32)\n') >>>a, *b = open('script2.py')# 3.X extended form>>>a, b('import sys\n', ['print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n']) >>>'y = 2\n' in open('script2.py')# Membership testFalse >>>'x = 2\n' in open('script2.py')True >>>L = [11, 22, 33, 44]# Slice assignment>>>L[1:3] = open('script2.py')>>>L[11, 'import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n', 44] >>>L = [11]>>>L.extend(open('script2.py'))# list.extend method>>>L[11, 'import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n']
>>>L = [11]>>>L.append(open('script2.py'))# list.append does not iterate>>>L[11, <_io.TextIOWrapper name='script2.py' mode='r' encoding='cp1252'>] >>>list(L[1])['import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n']
>>>set(open('script2.py')){'print(x ** 32)\n', 'import sys\n', 'print(sys.path)\n', 'x = 2\n'} >>>{line for line in open('script2.py')}{'print(x ** 32)\n', 'import sys\n', 'print(sys.path)\n', 'x = 2\n'} >>>{ix: line for ix, line in enumerate(open('script2.py'))}{0: 'import sys\n', 1: 'print(sys.path)\n', 2: 'x = 2\n', 3: 'print(x ** 32)\n'}
>>>{line for line in open('script2.py') if line[0] == 'p'}{'print(x ** 32)\n', 'print(sys.path)\n'} >>>{ix: line for (ix, line) in enumerate(open('script2.py')) if line[0] == 'p'}{1: 'print(sys.path)\n', 3: 'print(x ** 32)\n'}
>>>list(line.upper() for line in open('script2.py'))# See Chapter 20['IMPORT SYS\n', 'PRINT(SYS.PATH)\n', 'X = 2\n', 'PRINT(X ** 32)\n']
>>>sum([3, 2, 4, 1, 5, 0])# sum expects numbers only15 >>>any(['spam', '', 'ni'])True >>>all(['spam', '', 'ni'])False >>>max([3, 2, 5, 1, 4])5 >>>min([3, 2, 5, 1, 4])1
>>>max(open('script2.py'))# Line with max/min string value'x = 2\n' >>>min(open('script2.py'))'import sys\n'
>>>def f(a, b, c, d): print(a, b, c, d, sep='&')... >>>f(1, 2, 3, 4)1&2&3&4 >>>f(*[1, 2, 3, 4])# Unpacks into arguments1&2&3&4 >>> >>>f(*open('script2.py'))# Iterates by lines too!import sys &print(sys.path) &x = 2 &print(x ** 32)
>>>X = (1, 2)>>>Y = (3, 4)>>> >>>list(zip(X, Y))# Zip tuples: returns an iterable[(1, 3), (2, 4)] >>> >>>A, B = zip(*zip(X, Y))# Unzip a zip!>>>A(1, 2) >>>B(3, 4)
>>>zip('abc', 'xyz')# An iterable in Python 3.X (a list in 2.X)<zip object at 0x000000000294C308> >>>list(zip('abc', 'xyz'))# Force list of results in 3.X to display[('a', 'x'), ('b', 'y'), ('c', 'z')]
>>>Z = zip((1, 2), (3, 4))# Unlike 2.X lists, cannot index, etc.>>>Z[0]TypeError: 'zip' object is not subscriptable
>>>M = map(lambda x: 2 ** x, range(3))>>>for i in M: print(i)... 1 2 4 >>>for i in M: print(i)# Unlike 2.X lists, one pass only (zip too)... >>>
C:\code>c:\python33\python>>>R = range(10)# range returns an iterable, not a list>>>Rrange(0, 10) >>>I = iter(R)# Make an iterator from the range iterable>>>next(I)# Advance to next result0# What happens in for loops, comprehensions, etc.>>>next(I)1 >>>next(I)2 >>>list(range(10))# To force a list if required[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>len(R)# range also does len and indexing, but no others10 >>>R[0]0 >>>R[-1]9 >>>next(I)# Continue taking from iterator, where left off3 >>>I.__next__()# .next() becomes .__next__(), but use new next()4
>>>M = map(abs, (-1, 0, 1))# map returns an iterable, not a list>>>M<map object at 0x00000000029B75C0> >>>next(M)# Use iterator manually: exhausts results1# These do not support len() or indexing>>>next(M)0 >>>next(M)1 >>>next(M)StopIteration >>>for x in M: print(x)# map iterator is now empty: one pass only... >>>M = map(abs, (-1, 0, 1))# Make a new iterable/iterator to scan again>>>for x in M: print(x)# Iteration contexts auto call next()... 1 0 1 >>>list(map(abs, (-1, 0, 1)))# Can force a real list if needed[1, 0, 1]
>>>Z = zip((1, 2, 3), (10, 20, 30))# zip is the same: a one-pass iterator>>>Z<zip object at 0x0000000002951108> >>>list(Z)[(1, 10), (2, 20), (3, 30)] >>>for pair in Z: print(pair)# Exhausted after one pass... >>>Z = zip((1, 2, 3), (10, 20, 30))>>>for pair in Z: print(pair)# Iterator used automatically or manually... (1, 10) (2, 20) (3, 30) >>>Z = zip((1, 2, 3), (10, 20, 30))# Manual iteration (iter() not needed)>>>next(Z)(1, 10) >>>next(Z)(2, 20)
>>>filter(bool, ['spam', '', 'ni'])<filter object at 0x00000000029B7B70> >>>list(filter(bool, ['spam', '', 'ni']))['spam', 'ni']
>>>[x for x in ['spam', '', 'ni'] if bool(x)]['spam', 'ni'] >>>[x for x in ['spam', '', 'ni'] if x]['spam', 'ni']
>>>R = range(3)# range allows multiple iterators>>>next(R)TypeError: range object is not an iterator >>>I1 = iter(R)>>>next(I1)0 >>>next(I1)1 >>>I2 = iter(R)# Two iterators on one range>>>next(I2)0 >>>next(I1)# I1 is at a different spot than I22
>>>Z = zip((1, 2, 3), (10, 11, 12))>>>I1 = iter(Z)>>>I2 = iter(Z)# Two iterators on one zip>>>next(I1)(1, 10) >>>next(I1)(2, 11) >>>next(I2)# (3.X) I2 is at same spot as I1!(3, 12) >>>M = map(abs, (-1, 0, 1))# Ditto for map (and filter)>>>I1 = iter(M); I2 = iter(M)>>>print(next(I1), next(I1), next(I1))1 0 1 >>>next(I2)# (3.X) Single scan is exhausted!StopIteration >>>R = range(3)# But range allows many iterators>>>I1, I2 = iter(R), iter(R)>>>[next(I1), next(I1), next(I1)][0 1 2] >>>next(I2)# Multiple active scans, like 2.X lists0
>>>D = dict(a=1, b=2, c=3)>>>D{'a': 1, 'b': 2, 'c': 3} >>>K = D.keys()# A view object in 3.X, not a list>>>Kdict_keys(['a', 'b', 'c']) >>>next(K)# Views are not iterators themselvesTypeError: dict_keys object is not an iterator >>>I = iter(K)# View iterables have an iterator,>>>next(I)# which can be used manually,'a'# but does not support len(), index>>>next(I)'b' >>>for k in D.keys(): print(k, end=' ')# All iteration contexts use auto... a b c
>>>K = D.keys()>>>list(K)# Can still force a real list if needed['a', 'b', 'c'] >>>V = D.values()# Ditto for values() and items() views>>>Vdict_values([1, 2, 3]) >>>list(V)# Need list() to display or index as list[1, 2, 3] >>>V[0]TypeError: 'dict_values' object does not support indexing >>>list(V)[0]1 >>>list(D.items())[('a', 1), ('b', 2), ('c', 3)] >>>for (k, v) in D.items(): print(k, v, end=' ')... a 1 b 2 c 3
>>>D# Dictionaries still produce an iterator{'a': 1, 'b': 2, 'c': 3}# Returns next key on each iteration>>>I = iter(D)>>>next(I)'a' >>>next(I)'b' >>>for key in D: print(key, end=' ')# Still no need to call keys() to iterate...# But keys is an iterable in 3.X too!a b c
>>>D{'a': 1, 'b': 2, 'c': 3} >>>for k in sorted(D.keys()): print(k, D[k], end=' ')... a 1 b 2 c 3 >>>for k in sorted(D): print(k, D[k], end=' ')# "Best practice" key sorting... a 1 b 2 c 3
for loop uses the
iteration protocol to step through items in the
iterable object across which it is iterating. It first fetches an
iterator from the iterable by passing the object to iter, and then calls this iterator object’s
__next__ method in 3.X on each
iteration and catches the StopIteration exception to determine when to
stop looping. The method is named next in 2.X, and is run by the next built-in function in both 3.x and 2.X.
Any object that supports this model works in a for loop and in all other iteration
contexts. For some objects that are their own iterator, the initial
iter call is extraneous but
harmless.for loop task: collecting the results of
applying an expression to all items in an iterable object. It’s always
possible to translate a list comprehension to a for loop, and part of the list comprehension
expression looks like the header of a for loop syntactically.for loop; list comprehensions; the map built-in function; the in membership test expression; and the
built-in functions sorted, sum, any,
and all. This category also
includes the list and tuple built-ins, string join methods, and sequence assignments, all
of which use the iteration protocol (see answer #1) to step across
iterable objects one item at a time.for loop or
list comprehension, and let the iteration tool automatically scan one
line at a time by running the file’s next handler method on each iteration. This
approach is generally best in terms of coding simplicity, memory
space, and possibly execution speed requirements.1 Spoiler alert: the file iterator still appears to be slightly
faster than readlines and at
least 30% faster than the while
loop in both 2.7 and 3.3 on tests I’ve run with this chapter’s code
on a 1,000-line file (while is
twice as slow on 2.7). The usual benchmarking caveats apply — this is
true only for my Pythons, my computer, and my test file, and Python
3.X complicates such analyses by rewriting I/O libraries to support
Unicode text and be less system-dependent. Chapter 21 covers tools and techniques
you can use to time these loop statements on your own.
2 Technically speaking, the for loop calls the internal equivalent
of I.__next__, instead of the
next(I) used here, though there
is rarely any difference between the two. Your manual iterations
can generally use either call scheme.
| Form | Role |
|---|---|
>>>import sys>>>dir(sys)['__displayhook__',...more names omitted..., 'winver']
>>>len(dir(sys))# Number names in sys78 >>>len([x for x in dir(sys) if not x.startswith('__')])# Non __X names only69 >>>len([x for x in dir(sys) if not x[0] == '_'])# Non underscore names62
>>>dir([])['__add__', '__class__', '__contains__',...more..., 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort'] >>>dir('')['__add__', '__class__', '__contains__',...more..., 'split', 'splitlines', 'startswith','strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
>>>len(dir([])), len([x for x in dir([]) if not x.startswith('__')])(45, 11) >>>len(dir('')), len([x for x in dir('') if not x.startswith('__')])(76, 44)
>>>[a for a in dir(list) if not a.startswith('__')]['append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort'] >>>[a for a in dir(dict) if not a.startswith('__')]['clear', 'copy', 'fromkeys', 'get', 'items', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values']
>>>def dir1(x): return [a for a in dir(x) if not a.startswith('__')]# See Part IV... >>>dir1(tuple)['count', 'index']
>>>dir(str) == dir('')# Same result, type name or literalTrue >>>dir(list) == dir([])True
"""
Module documentation
Words Go Here
"""
spam = 40
def square(x):
"""
function documentation
can we have your liver then?
"""
return x ** 2 # square
class Employee:
"class documentation"
pass
print(square(4))
print(square.__doc__)>>>import docstrings16 function documentation can we have your liver then? >>>print(docstrings.__doc__)Module documentation Words Go Here >>>print(docstrings.square.__doc__)function documentation can we have your liver then? >>>print(docstrings.Employee.__doc__)class documentation
>>>import sys>>>print(sys.__doc__)This module provides access to some objects used or maintained by the interpreter and to functions that interact strongly with the interpreter. Dynamic objects: argv -- command line arguments; argv[0] is the script pathname if known path -- module search path; path[0] is the script directory, else '' modules -- dictionary of loaded modules...more text omitted...
>>> print(sys.getrefcount.__doc__)
getrefcount(object) -> integer
Return the reference count of object. The count returned is generally
one higher than you might expect, because it includes the (temporary)
reference as an argument to getrefcount().>>>print(int.__doc__)int(x[, base]) -> integer Convert a string or number to an integer, if possible. A floating point argument will be truncated towards zero (this does not include a...more text omitted...>>>print(map.__doc__)map(func, *iterables) --> map object Make an iterator that computes the function using arguments from each of the iterables. Stops when the shortest iterable is exhausted.
>>>import sys>>>help(sys.getrefcount)Help on built-in function getrefcount in module sys: getrefcount(...) getrefcount(object) -> integer Return the reference count of object. The count returned is generally one higher than you might expect, because it includes the (temporary) reference as an argument to getrefcount().
>>>help(sys)Help on built-in module sys: NAME sys MODULE REFERENCE http://docs.python.org/3.3/library/sys...more omitted...DESCRIPTION This module provides access to some objects used or maintained by the interpreter and to functions that interact strongly with the interpreter....more omitted...FUNCTIONS __displayhook__ = displayhook(...) displayhook(object) -> None...more omitted...DATA __stderr__ = <_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp4... __stdin__ = <_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp437... __stdout__ = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp4......more omitted...FILE (built-in)
>>>help(dict)Help on class dict in module builtins: class dict(object) | dict() -> new empty dictionary. | dict(mapping) -> new dictionary initialized from a mapping object's...more omitted...>>>help(str.replace)Help on method_descriptor: replace(...) S.replace (old, new[, count]) -> str Return a copy of S with all occurrences of substring...more omitted...>>>help(''.replace)...similar to prior result...>>>help(ord)Help on built-in function ord in module builtins: ord(...) ord(c) -> integer Return the integer ordinal of a one-character string.
>>>import docstrings>>>help(docstrings.square)Help on function square in module docstrings: square(x) function documentation can we have your liver then? >>>help(docstrings.Employee)Help on class Employee in module docstrings: class Employee(builtins.object) | class documentation |...more omitted...>>>help(docstrings)Help on module docstrings: NAME docstrings DESCRIPTION Module documentation Words Go Here CLASSES builtins.object Employee class Employee(builtins.object) | class documentation |...more omitted...FUNCTIONS square(x) function documentation can we have your liver then? DATA spam = 40 FILE c:\code\docstrings.py
c:\code>python -m pydoc -bServer ready at http://localhost:62135/ Server commands: [b]rowser, [q]uit server>qServer stopped c:\code>py −3 -m pydoc -bServer ready at http://localhost:62144/ Server commands: [b]rowser, [q]uit server>qServer stopped c:\code>C:\python33\python -m pydoc -bServer ready at http://localhost:62153/ Server commands: [b]rowser, [q]uit server>qServer stopped
c:\code>py −3 -m pydoc timeit# Command-line text helpc:\code>py −3>>>help("timeit")# Interactive prompt text help
c:\code>c:\python32\python -m pydoc -g# Explicit Python pathc:\code>py −3.2 -m pydoc -g# Windows 3.3+ launcher version
c:\code>set PYTHONPATH=.;%PYTHONPATH%c:\code>py −3.2 -m pydoc -g
: at the
end of compound statement headers — the first line of an if, while, for, etc. You’ll probably forget at first (I
did, and so have most of my roughly 4,000 Python students over the
years), but you can take some comfort from the fact that it will soon
become an unconscious habit.... prompt (or in IDLE)
until you’re really done. This also means you can’t paste multiline
code at this prompt; it must run one full statement at a time.if and while headers (e.g., if (X==1):). You can, if you like (any
expression can be enclosed in parentheses), but they are fully
superfluous in this context. Also, do not terminate all your
statements with semicolons; it’s technically legal to do this in
Python as well, but it’s totally useless unless you’re placing more
than one statement on a single line (the end of a line normally
terminates a statement). And remember, don’t embed assignment
statements in while loop tests, and
don’t use {} around blocks (indent
your nested code blocks consistently instead).for loops instead
of while or range. Another reminder: a simple for loop (e.g., for
x in seq:) is almost always simpler to code and often
quicker to run than a while- or
range-based counter loop. Because
Python handles indexing internally for a simple for, it can sometimes be faster than the
equivalent while, though this can
vary per code and Python. For code simplicity alone, though, avoid the
temptation to count things in Python!a = b = []), as well as in an augmented
assignment (a += [1, 2]). In both
cases, in-place changes may impact other variables. See Chapter 11 for details if
you’ve forgotten why this is true.list.append and list.sort methods introduced in Chapter 8 do not return values (other than
None), so you should call them
without assigning the result. It’s not uncommon for beginners to say
something like mylist =
mylist.append(X) to try to get the result of an append, but what this actually does is
assign mylist to None, not to the modified list (in fact,
you’ll lose your reference to the list altogether).for k in
D.keys().sort():. This almost works — the keys method builds a keys list, and the
sort method orders it — but because
the sort method returns None, the loop fails because it is
ultimately a loop over None (a
nonsequence). This fails even sooner in Python 3.X, because dictionary
keys are views, not lists! To code this correctly, either use the
newer sorted built-in function,
which returns the sorted list, or split the method calls out to
statements: Ks = list(D.keys()),
then Ks.sort(), and finally,
for k in Ks:. This, by the way, is
one case where you may still want to call the keys method explicitly for looping, instead
of relying on the dictionary iterators — iterators do not sort.function(), not function). In the next part of this book,
we’ll learn that functions are simply objects that have a special
operation — a call that you trigger with the parentheses. They can be
referenced like any other object without triggering a call.file.close to close a file, rather than
file.close(). Because it’s legal to
reference a function without calling it, the first version with no
parentheses succeeds silently, but it does not close the file!import statements — say import mod, not import mod.py. We discussed module basics in
Chapter 3 and will continue studying
modules in Part V. Because modules
may have other extensions besides .py (.pyc, for instance), hardcoding a
particular extension is not only illegal syntax, it doesn’t make
sense. Python picks an extension automatically, and any
platform-specific directory path syntax comes from module search path
settings, not the import
statement.__doc__ attribute, by passing it to PyDoc’s
help function, and by selecting
modules in PyDoc’s HTML-based user interfaces — either the -g GUI client mode in Python 3.2 and
earlier, or the -b all-browser mode
in Python 3.2 and later (and required as of 3.3). Both run a
client/server system that displays documentation in a popped-up web
browser. PyDoc can also be run to save a module’s documentation in an
HTML file for later viewing or printing.dir(X) function
returns a list of all the attributes attached to any object. A list
comprehension of the form [a for a in dir(X)
if not a.startswith('__')] can be used to filter out
internals names with underscores (we’ll learn how to wrap this in a
function in the next part of the book to make it easier to
use).-b command-line switch; the
top-level start page displayed in a web browser in this newer mode has
the same index page listing all available modules.for loops.for loop that
prints the ASCII code of each character in a string named S. Use the built-in function ord(character) to convert each character
to an ASCII integer. This function technically returns a Unicode
code point in Python 3.X, but if you restrict its content to ASCII
characters, you’ll get back ASCII codes. (Test it interactively to
see how it works.)map(ord,
S) have a similar effect? How about [ord(c) for c in S]? Why? (Hint: see
Chapter 14.)for i in range(50):
print('hello %d\n\a' % i)for
loop that prints a dictionary’s items in sorted (ascending) order.
(Hint: use the dictionary keys and
list sort methods, or the newer
sorted built-in function.)while
loop and found flag to search a
list of powers of 2 for the value of 2 raised to the fifth power (32).
It’s stored in a module file called power.py.L = [1, 2, 4, 8, 16, 32, 64]
X = 5
found = False
i = 0
while not found and i < len(L):
if 2 ** X == L[i]:
found = True
else:
i = i+1
if found:
print('at index', i)
else:
print(X, 'not found')
C:\book\tests> python power.py
at index 5while loop else clause to eliminate the found flag and final if statement.for loop with an else clause, to eliminate the explicit
list-indexing logic. (Hint: to get the index of an item, use the
list index method — L.index(X) returns the offset of the
first X in list L.)in operator
membership expression. (See Chapter 8 for more details, or type this
to test: 2 in [1,2,3].)for loop
and the list append method to
generate the powers-of-2 list (L) instead of hardcoding a list
literal.2 ** X expression outside the
loops? How would you code that?map(function, list) tool that can
generate a powers-of-2 list, too: map(lambda x: 2 ** x, range(7)). Try
typing this code interactively; we’ll meet lambda more formally in the next part of
this book, especially in Chapter 19. Would a list comprehension
help here (see Chapter 14)?fc on Windows)
with the original pydoc.py in 3.3
(also included, lest it change radically in 3.4 as the sidebar
describes). If PyDoc is more easily customized by the time you read
these words, customize colors per its current convention instead; if
this involves changing a CSS file, let’s hope the procedure will be
well documented in Python’s manuals.1 Note that asking for help on an actual string
object directly (e.g., help('')) doesn’t work in recent Pythons:
you usually get no help, because strings are interpreted
specially — as a request for help on an unimported module, for
instance (see earlier). You must use the str type name in this context, though both
other types of actual objects (help([])) and string method names
referenced through actual objects (help(''.join)) work fine (at least in
Python 3.3 — this has been prone to change over time). There is also
an interactive help mode, which you start by typing just help().
| Statement or expression | Examples |
|---|---|
myfunc('spam', 'eggs', meat=ham, *rest) | |
def printer(message):
print('Hello ' + message) | |
def adder(a, b=1, *c):
return a + b + c[0] | |
x = 'old'
def changer():
global x; x = 'new' | |
def outer():
x = 'old'
def changer():
nonlocal x; x = 'new' | |
def squares(x):
for i in range(x): yield i ** 2 | |
funcs = [lambda x: x**2, lambda x: x**3] |
As in most programming languages, Python functions are the simplest way to package logic you may wish to use in more than one place and more than one time. Up until now, all the code we’ve been writing has run immediately. Functions allow us to group and generalize code to be used arbitrarily many times later. Because they allow us to code an operation in a single place and use it in many places, Python functions are the most basic factoring tool in the language: they allow us to reduce code redundancy in our programs, and thereby reduce maintenance effort.
Functions also provide a tool for splitting systems into pieces that have well-defined roles. For instance, to make a pizza from scratch, you would start by mixing the dough, rolling it out, adding toppings, baking it, and so on. If you were programming a pizza-making robot, functions would help you divide the overall “make pizza” task into chunks — one function for each subtask in the process. It’s easier to implement the smaller tasks in isolation than it is to implement the entire process at once. In general, functions are about procedure — how to do something, rather than what you’re doing it to. We’ll see why this distinction matters in Part VI, when we start making new objects with classes.
def is executable code. Python functions
are written with a new statement, the def. Unlike functions in compiled languages
such as C, def is an executable
statement — your function does not exist until Python reaches and runs
the def. In fact, it’s legal (and
even occasionally useful) to nest def statements inside if statements, while loops, and even other defs. In typical operation, def statements are coded in module files and
are naturally run to generate functions when the module file they
reside in is first imported.def creates an object and assigns it to a name.
When Python reaches and runs a def
statement, it generates a new function object and assigns it to the
function’s name. As with all assignments, the function name becomes a
reference to the function object. There’s nothing magic about the name
of a function — as you’ll see, the function object can be assigned to
other names, stored in a list, and so on. Function objects may also
have arbitrary user-defined attributes attached
to them to record data.lambda creates an object but returns it as a result.
Functions may also be created with the lambda expression, a feature that allows us
to in-line function definitions in places where a
def statement won’t work
syntactically. This is a more advanced concept that we’ll defer until
Chapter 19.return sends a result object back to the caller.
When a function is called, the caller stops until the function
finishes its work and returns control to the caller. Functions that
compute a value send it back to the caller with a return statement; the returned value becomes
the result of the function call. A return without a value simply returns to the
caller (and sends back None, the
default result).yield sends a result object back to the caller, but remembers
where it left off. Functions known as generators may also use
the yield statement to send back a
value and suspend their state such that they may be resumed later, to
produce a series of results over time. This is another advanced topic
covered later in this part of the book.global declares module-level variables that are to be
assigned. By default, all names assigned in a function are local to that function and
exist only while the function runs. To assign a name in the enclosing
module, functions need to list it in a global statement. More generally, names are
always looked up in scopes — places where variables
are stored — and assignments bind names to scopes.nonlocal declares enclosing function variables that are to be
assigned. Similarly, the nonlocal statement added in Python 3.X allows a function to assign a name
that exists in the scope of a syntactically enclosing def statement. This allows enclosing
functions to serve as a place to retain
state — information remembered between function
calls — without using shared global names.name=value keyword
syntax, and unpack arbitrarily many arguments to send with *pargs and
**kargs
starred-argument notation. Function definitions
use the same two forms to specify argument defaults, and collect
arbitrarily many arguments received.defname(arg1,arg2,...argN):statements
defname(arg1,arg2,...argN): ... returnvalue
if test:
def func(): # Define func this way
...
else:
def func(): # Or else this way
...
...
func() # Call the version selected and builtothername = func# Assign function objectothername()# Call func again
def func(): ...# Create function objectfunc()# Call objectfunc.attr = value# Attach attributes
>>>def times(x, y):# Create and assign function...return x * y# Body executed when called...
>>>times(2, 4)# Arguments in parentheses8
>>>x = times(3.14, 4)# Save the result object>>>x12.56
>>>times('Ni', 4)# Functions are "typeless"'NiNiNiNi'
def intersect(seq1, seq2):
res = [] # Start empty
for x in seq1: # Scan seq1
if x in seq2: # Common item?
res.append(x) # Add to end
return res>>>s1 = "SPAM">>>s2 = "SCAM">>>intersect(s1, s2)# Strings['S', 'A', 'M']
>>> [x for x in s1 if x in s2]
['S', 'A', 'M']>>>x = intersect([1, 2, 3], (1, 4))# Mixed types>>>x# Saved result object[1]
res is obviously assigned,
so it is a local variable.seq1 and seq2 are, too.for loop assigns items
to a variable, so the name x is
also local.def statement; this statement creates a
function object and assigns it the function’s name. This normally
happens when the enclosing module file is imported by another module
(recall that imports run the code in a file from top to bottom,
including any defs), but it can
also occur when a def is typed
interactively or nested in other statements, such as ifs.None
object by default if the control flow falls off the end of the
function body without running into a return statement. Such functions are usually
called with expression statements, as assigning their None results to variables is generally
pointless. A return statement with
no expression in it also returns None.1 This polymorphic behavior has in recent years come to also be known as duck typing — the essential idea being that your code is not supposed to care if an object is a duck, only that it quacks. Anything that quacks will do, duck or not, and the implementation of quacks is up to the object, a principle which will become even more apparent when we study classes in Part VI. Graphic metaphor to be sure, though this is really just a new label for an older idea, and use cases for quacking software would seem limited in the tangible world (he says, bracing for emails from militant ornithologists...).
2 This code will always work if we intersect files’ contents
obtained with file.readlines().
It may not work to intersect lines in open input files directly,
though, depending on the file object’s implementation of the
in operator or general iteration.
Files must generally be rewound (e.g., with a file.seek(0) or another open) after they have been read to
end-of-file once, and so are single-pass iterators. As we’ll see in
Chapter 30 when we study operator
overloading, objects implement the in operator either by providing the
specific __contains__ method or
by supporting the general iteration protocol with the __iter__ or older __getitem__ methods; classes can code
these methods arbitrarily to define what iteration means for their
data.
def
can only be seen by the code within that def. You cannot even refer to such names
from outside the function.def
do not clash with variables outside the def, even if the same names are used
elsewhere. A name X assigned
outside a given def (i.e., in a
different def or at the top level
of a module file) is a completely different variable from a name
X assigned inside that def.def, it is local to
that function.def, it is nonlocal to
nested functions.defs, it is global to
the entire file.X = 99# Global (module) scope Xdef func(): X = 88# Local (function) scope X: a different variable
global statement inside the function. If you need to assign a name that lives in
an enclosing def, as of Python
3.X you can do so by declaring it in a nonlocal
statement.def statement;
globals that live in the enclosing module’s
namespace; or built-ins in the predefined built-ins module Python provides.def statement
(and lambda expression) as defining a new local scope, but the local scope
actually corresponds to a function call.
Because Python allows functions to call themselves to loop — an
advanced technique known as recursion and noted briefly in
Chapter 9 when we
explored comparisons — each active call receives its own copy of the
function’s local variables. Recursion is useful in functions we
write as well, to process structures whose shapes can’t be predicted
ahead of time; we’ll explore it more fully in Chapter 19.global
and nonlocal statements map
assigned names to enclosing module and function scopes,
respectively.defs and lambdas, then the global (G) scope, and then
the built-in (B) scope — and stops at the first
place the name is found. If the name is not found during this
search, Python reports an error.X used to refer to the current iteration item in a comprehension
expression such as [X for X in
I]. Because they might clash with other names and
reflect internal state in generators, in 3.X, such variables are
local to the expression itself in all comprehension forms:
generator, list, set, and dictionary. In 2.X, they are local to
generator expressions and set and dictionary comprehensions, but
not to list comprehensions that map their names to the scope
outside the expression. By contrast, for loop statements never localize their
variables to the statement block in any Python. See Chapter 20 for more details and
examples.X used to reference the raised exception in a try statement handler clause such as
except E as X. Because they
might defer garbage collection’s memory recovery, in 3.X, such
variables are local to that except block, and in fact are removed
when the block is exited (even if you’ve used it earlier in your
code!). In 2.X, these variables live on after the try statement. See Chapter 34 for additional
information.# Global scopeX = 99# X and func assigned in module: globaldef func(Y):# Y and Z assigned in function: locals# Local scopeZ = X + Y# X is a globalreturn Z func(1)# func in module: result=100
X, funcXis global because it’s assigned at the top level of the module file; it can be referenced inside the function as a simple unqualified variable without being declared global.funcis global for the same reason; thedefstatement assigns a function object to the namefuncat the top level of the module.
Y, ZYandZare local to the function (and exist only while the function runs) because they are both assigned values in the function definition:Zby virtue of the=statement, andYbecause arguments are always passed by assignment.
>>>import builtins>>>dir(builtins)['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning',...many more names omitted...'ord', 'pow', 'print', 'property', 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']
>>>zip# The normal way<class 'zip'> >>>import builtins# The hard way: for customizations>>>builtins.zip<class 'zip'> >>>zip is builtins.zip# Same object, different lookupsTrue
def hider():
open = 'spam' # Local variable, hides built-in here
...
open('data.txt') # Error: this no longer opens a file in this scope!>>>open = 99# Assign in global scope, hides built-in here too
>>> len(dir(builtins)), len([x for x in dir(builtins) if not x.startswith('__')])
(148, 142)X = 88# Global Xdef func(): X = 99# Local X: hides global, but we want this herefunc() print(X)# Prints 88: unchanged
X = 88# Global Xdef func(): global X X = 99# Global X: outside deffunc() print(X)# Prints 99
y, z = 1, 2# Global variables in moduledef all_global(): global x# Declare globals assignedx = y + z# No need to declare y, z: LEGB rule
X = 99
def func1():
global X
X = 88
def func2():
global X
X = 77# first.pyX = 99# This code doesn't know about second.py# second.pyimport first print(first.X)# OK: references a name in another filefirst.X = 88# But changing it can be too subtle and implicit
# first.pyX = 99 def setX(new):# Accessors make external changes explicitglobal X# And can manage access in a single placeX = new# second.pyimport first first.setX(88)# Call the function instead of changing directly
# thismod.pyvar = 99# Global variable == module attributedef local(): var = 0# Change local vardef glob1(): global var# Declare global (normal)var += 1# Change global vardef glob2(): var = 0# Change local varimport thismod# Import myselfthismod.var += 1# Change global vardef glob3(): var = 0# Change local varimport sys# Import system tableglob = sys.modules['thismod']# Get module object (or use __name__)glob.var += 1# Change global vardef test(): print(var) local(); glob1(); glob2(); glob3() print(var)
>>>import thismod>>>thismod.test()99 102 >>>thismod.var102
X) looks for the name X first in the
current local scope (function); then in the local scopes of any
lexically enclosing functions in your source code, from inner to
outer; then in the current global scope (the module file); and
finally in the built-in scope (the module builtins). global declarations make the search begin
in the global (module file) scope instead.X = value) creates or changes the name X
in the current local scope, by default. If X is declared global
within the function, the assignment creates or changes the name
X in the enclosing module’s scope
instead. If, on the other hand, X
is declared nonlocal within the function in 3.X
(only), the assignment changes the name X in the closest enclosing function’s
local scope.X = 99# Global scope name: not useddef f1(): X = 88# Enclosing def localdef f2(): print(X)# Reference made in nested deff2() f1()# Prints 88: enclosing def local
def f1():
X = 88
def f2():
print(X) # Remembers X in enclosing def scope
return f2 # Return f2 but don't call it
action = f1() # Make, return function
action() # Call it now: prints 88>>>def maker(N):def action(X):# Make and return actionreturn X ** N# action retains N from enclosing scopereturn action
>>>f = maker(2)# Pass 2 to argument N>>>f<function maker.<locals>.action at 0x0000000002A4A158>
>>>f(3)# Pass 3 to X, N remembers 2: 3 ** 29 >>>f(4)# 4 ** 216
>>>g = maker(3)# g remembers 3, f remembers 2>>>g(4)# 4 ** 364 >>>f(4)# 4 ** 216
>>>def maker(N):return lambda X: X ** N# lambda functions retain state too>>>h = maker(3)>>>h(4)# 4 ** 3 again64
def f1():
x = 88
def f2(x=x): # Remember enclosing scope X with defaults
print(x)
f2()
f1() # Prints 88>>>def f1():x = 88# Pass x along instead of nestingf2(x)# Forward reference OK>>>def f2(x):print(x)# Flat is still often better than nested!>>>f1()88
def func():
x = 4
action = (lambda n: x ** n) # x remembered from enclosing def
return action
x = func()
print(x(2)) # Prints 16, 4 ** 2def func():
x = 4
action = (lambda n, x=x: x ** n) # Pass x in manually
return action>>>def makeActions():acts = []for i in range(5):# Tries to remember each iacts.append(lambda x: i ** x)# But all remember same last i!return acts>>>acts = makeActions()>>>acts[0]<function makeActions.<locals>.<lambda> at 0x0000000002A4A400>
>>>acts[0](2)# All are 4 ** 2, 4=value of last i16 >>>acts[1](2)# This should be 1 ** 2 (1)16 >>>acts[2](2)# This should be 2 ** 2 (4)16 >>>acts[4](2)# Only this should be 4 ** 2 (16)16
>>>def makeActions():acts = []for i in range(5):# Use defaults insteadacts.append(lambda x, i=i: i ** x)# Remember current ireturn acts>>>acts = makeActions()>>>acts[0](2)# 0 ** 20 >>>acts[1](2)# 1 ** 21 >>>acts[2](2)# 2 ** 24 >>>acts[4](2)# 4 ** 216
>>>def f1():x = 99def f2():def f3():print(x)# Found in f1's local scope!f3()f2()>>>f1()99
def func():
nonlocal name1, name2, ... # OK here
>>> nonlocal X
SyntaxError: nonlocal declaration not allowed at module levelglobal makes scope
lookup begin in the enclosing module’s scope and
allows names there to be assigned. Scope lookup continues on to the
built-in scope if the name does not exist in the module, but
assignments to global names always create or change them in the
module’s scope.nonlocal restricts scope
lookup to just enclosing defs,
requires that the names already exist there, and allows them to be
assigned. Scope lookup does not continue on to the global or
built-in scopes.C:\code>c:\python33\python>>>def tester(start):state = start# Referencing nonlocals works normallydef nested(label):print(label, state)# Remembers state in enclosing scopereturn nested>>>F = tester(0)>>>F('spam')spam 0 >>>F('ham')ham 0
>>>def tester(start):state = startdef nested(label):print(label, state)state += 1# Cannot change by default (never in 2.X)return nested>>>F = tester(0)>>>F('spam')UnboundLocalError: local variable 'state' referenced before assignment
>>>def tester(start):state = start# Each call gets its own statedef nested(label):nonlocal state# Remembers state in enclosing scopeprint(label, state)state += 1# Allowed to change it if nonlocalreturn nested>>>F = tester(0)>>>F('spam')# Increments state on each callspam 0 >>>F('ham')ham 1 >>>F('eggs')eggs 2
>>>G = tester(42)# Make a new tester that starts at 42>>>G('spam')spam 42 >>>G('eggs')# My state information updated to 43eggs 43 >>>F('bacon')# But F's is where it left off: at 3bacon 3# Each call has different state information
>>>def tester(start):def nested(label):nonlocal state# Nonlocals must already exist in enclosing def!state = 0print(label, state)return nestedSyntaxError: no binding for nonlocal 'state' found >>>def tester(start):def nested(label):global state# Globals don't have to exist yet when declaredstate = 0# This creates the name in the module nowprint(label, state)return nested>>>F = tester(0)>>>F('abc')abc 0 >>>state0
>>>spam = 99>>>def tester():def nested():nonlocal spam# Must be in a def, not the module!print('Current=', spam)spam += 1return nestedSyntaxError: no binding for nonlocal 'spam' found
>>>def tester(start):state = start# Each call gets its own statedef nested(label):nonlocal state# Remembers state in enclosing scopeprint(label, state)state += 1# Allowed to change it if nonlocalreturn nested>>>F = tester(0)>>>F('spam')# State visible within closure onlyspam 0 >>>F.stateAttributeError: 'function' object has no attribute 'state'
>>>def tester(start):global state# Move it out to the module to change itstate = start# global allows changes in module scopedef nested(label):global stateprint(label, state)state += 1return nested>>>F = tester(0)>>>F('spam')# Each call increments shared global statespam 0 >>>F('eggs')eggs 1
>>>G = tester(42)# Resets state's single copy in global scope>>>G('toast')toast 42 >>>G('bacon')bacon 43 >>>F('ham')# But my counter has been overwritten!ham 44
>>>class tester:# Class-based alternative (see Part VI)def __init__(self, start):# On object construction,self.state = start# save state explicitly in new objectdef nested(self, label):print(label, self.state)# Reference state explicitlyself.state += 1# Changes are always allowed>>>F = tester(0)# Create instance, invoke __init__>>>F.nested('spam')# F is passed to selfspam 0 >>>F.nested('ham')ham 1
>>>G = tester(42)# Each instance gets new copy of state>>>G.nested('toast')# Changing one does not impact otherstoast 42 >>>G.nested('bacon')bacon 43 >>>F.nested('eggs')# F's state is where it left offeggs 2 >>>F.state# State may be accessed outside class3
>>>class tester:def __init__(self, start):self.state = startdef __call__(self, label):# Intercept direct instance callsprint(label, self.state)# So .nested() not requiredself.state += 1>>>H = tester(99)>>>H('juice')# Invokes __call__juice 99 >>>H('pancakes')pancakes 100
>>>def tester(start):def nested(label):print(label, nested.state)# nested is in enclosing scopenested.state += 1# Change attr, not nested itselfnested.state = start# Initial state after func definedreturn nested>>>F = tester(0)>>>F('spam')# F is a 'nested' with state attachedspam 0 >>>F('ham')ham 1 >>>F.state# Can access state outside functions too2
>>>G = tester(42)# G has own state, doesn't overwrite F's>>>G('eggs')eggs 42 >>>F('ham')ham 2 >>>F.state# State is accessible and per-call3 >>>G.state43 >>>F is G# Different function objectsFalse
def tester(start):
def nested(label):
print(label, state[0]) # Leverage in-place mutable change
state[0] += 1 # Extra syntax, deep magic?
state = [start]
return nested>>>X = 'Spam'>>>def func():print(X)>>>func()
>>>X = 'Spam'>>>def func():X = 'NI!'>>>func()>>>print(X)
>>>X = 'Spam'>>>def func():X = 'NI'print(X)>>>func()>>>print(X)
>>>X = 'Spam'>>>def func():global XX = 'NI'>>>func()>>>print(X)
>>>X = 'Spam'>>>def func():X = 'NI'def nested():print(X)nested()>>>func()>>>X
>>>def func():X = 'NI'def nested():nonlocal XX = 'Spam'nested()print(X)>>>func()
'Spam',
because the function references a global variable in the enclosing
module (because it is not assigned in the function, it is considered
global).'Spam'
again because assigning the variable inside the function makes it a
local and effectively hides the global of the same name. The print statement finds the variable unchanged
in the global (module) scope.'NI' on one line
and 'Spam' on another, because the
reference to the variable within the function finds the assigned local
and the reference in the print
statement finds the global.'NI'
because the global declaration forces the variable assigned inside the
function to refer to the variable in the enclosing global
scope.'NI' on one line and 'Spam' on another, because the print statement in the nested function finds
the name in the enclosing function’s local scope, and the display at
the end finds the variable in the global scope.'Spam',
because the nonlocal statement
(available in Python 3.X but not 2.X) means that the assignment to
X inside the nested function
changes X in the enclosing
function’s local scope. Without this statement, this assignment would
classify X as local to the nested
function, making it a different variable; the code would then print
'NI' instead.1 The scope lookup rule was called the “LGB rule” in the first
edition of this book. The enclosing def “E” layer was added later in Python to
obviate the task of passing in enclosing scope names explicitly with
default arguments — a topic usually of marginal interest to Python
beginners that we’ll defer until later in this chapter. Since this
scope is now addressed by the nonlocal statement in Python 3.X, the
lookup rule might be better named “LNGB” today, but backward
compatibility matters in books, too. The present form of this
acronym also does not account for the newer obscure scopes of some
comprehensions and exception handlers, but acronyms longer than four
letters tend to defeat their purpose!
2 Multithreading runs function calls in
parallel with the rest of the program and is supported by Python’s
standard library modules _thread,
threading, and queue (thread, threading, and Queue in Python 2.X). Because all threaded
functions run in the same process, global scopes often serve as one
form of shared memory between them (threads may share both names in
global scopes, as well as objects in a process’s memory space).
Threading is commonly used for long-running tasks in GUIs, to
implement nonblocking operations in general and to maximize CPU
capacity. It is also beyond this book’s scope; see the Python
library manual, as well as the follow-up texts listed in the preface
(such as O’Reilly’s Programming
Python), for more details.
3 In the section “Function Gotchas”, we’ll
also see that there is a similar issue with using mutable objects
like lists and dictionaries for default arguments (e.g., def f(a=[])) — because defaults are
implemented as single objects attached to functions, mutable
defaults retain state from call to call, rather then being
initialized anew on each call. Depending on whom you ask, this is
either considered a feature that supports another way to implement
state retention, or a strange corner of the language; more on this
at the end of Chapter 21.
L = [1, 2] changer(X,L[:])# Pass a copy, so our 'L' does not change
def changer(a, b):
b = b[:] # Copy input list so we don't impact caller
a = 2
b[0] = 'spam' # Changes our list copy onlyL = [1, 2] changer(X,tuple(L))# Pass a tuple, so changes are errors
>>>def multiple(x, y):x = 2# Changes local names onlyy = [3, 4]return x, y# Return multiple new values in a tuple>>>X = 1>>>L = [1, 2]>>>X, L = multiple(X, L)# Assign results to caller's names>>>X, L(2, [3, 4])
def f((a, (b, c))):
def f(T): (a, (b, c)) = T
Functions can use special arguments preceded with one or two*characters to collect an arbitrary number of possibly extra arguments. This feature is often referred to as varargs, after a variable-length argument list tool in the C language; in Python, the arguments are collected in a normal object.
Callers can also use the*syntax to unpack argument collections into separate arguments. This is the inverse of a*in a function header — in the header it means collect arbitrarily many arguments, while in the call it means unpack arbitrarily many arguments, and pass them individually as discrete values.
In Python 3.X (but not 2.X), functions can also specify arguments that must be passed by name with keyword arguments, not by position. Such arguments are typically used to define configuration options in addition to actual arguments.
| Syntax | Location | Interpretation |
|---|---|---|
def func(*, name=value) |
name=value form tells Python to
match by name to arguments instead; these are called
keyword arguments. Using a *iterable or **dict in a call allows us to package up
arbitrarily many positional or keyword objects in sequences (and
other iterables) and dictionaries, respectively, and unpack them as
separate, individual arguments when they are passed to the
function.name is matched
by position or name depending on how the caller passes it, but the
name=value form specifies a
default value. The *name form collects any extra unmatched
positional arguments in a tuple, and the **name form collects extra keyword
arguments in a dictionary. In Python 3.X, any normal or defaulted
argument names following a *name
or a bare * are
keyword-only arguments and must be passed by
keyword in calls.print
function, but they are more general — keywords allow us to label any
argument with its name, to make calls more informational.value); followed by a combination of any
keyword arguments (name=value)
and the *iterable form; followed
by the **dict form.name); followed by any default arguments
(name=value); followed by the
*name (or * in 3.X) form; followed by any name or name=value keyword-only arguments (in
3.X); followed by the **name
form.*name tuple.**name dictionary.>>>def f(a, b, c): print(a, b, c)>>>f(1, 2, 3)1 2 3
>>> f(c=3, b=2, a=1)
1 2 3>>>f(1, c=3, b=2)# a gets 1 by position, b and c passed by name1 2 3
func(name='Bob', age=40, job='dev')
>>>def f(a, b=2, c=3): print(a, b, c)# a required, b and c optional
>>>f(1)# Use defaults1 2 3 >>>f(a=1)1 2 3
>>>f(1, 4)# Override defaults1 4 3 >>>f(1, 4, 5)1 4 5
>>>f(1, c=6)# Choose defaults1 2 6
def func(spam, eggs, toast=0, ham=0):# First 2 requiredprint((spam, eggs, toast, ham)) func(1, 2)# Output: (1, 2, 0, 0)func(1, ham=1, eggs=0)# Output: (1, 0, 0, 1)func(spam=1, eggs=0)# Output: (1, 0, 0, 0)func(toast=1, eggs=2, spam=3)# Output: (3, 2, 1, 0)func(1, 2, 3, 4)# Output: (1, 2, 3, 4)
>>> def f(*args): print(args)>>>f()() >>>f(1)(1,) >>>f(1, 2, 3, 4)(1, 2, 3, 4)
>>>def f(**args): print(args)>>>f(){} >>>f(a=1, b=2){'a': 1, 'b': 2}
>>>def f(a, *pargs, **kargs): print(a, pargs, kargs)>>>f(1, 2, 3, x=1, y=2)1 (2, 3) {'y': 2, 'x': 1}
>>>def func(a, b, c, d): print(a, b, c, d)>>>args = (1, 2)>>>args += (3, 4)>>>func(*args)# Same as func(1, 2, 3, 4)1 2 3 4
>>>args = {'a': 1, 'b': 2, 'c': 3}>>>args['d'] = 4>>>func(**args)# Same as func(a=1, b=2, c=3, d=4)1 2 3 4
>>>func(*(1, 2), **{'d': 4, 'c': 3})# Same as func(1, 2, d=4, c=3)1 2 3 4 >>>func(1, *(2, 3), **{'d': 4})# Same as func(1, 2, 3, d=4)1 2 3 4 >>>func(1, c=3, *(2,), **{'d': 4})# Same as func(1, 2, c=3, d=4)1 2 3 4 >>>func(1, *(2, 3), d=4)# Same as func(1, 2, 3, d=4)1 2 3 4 >>>func(1, *(2,), c=3, **{'d':4})# Same as func(1, 2, c=3, d=4)1 2 3 4
ifsometest: action, args = func1, (1,)# Call func1 with one arg in this caseelse: action, args = func2, (1, 2, 3)# Call func2 with three args here...etc...action(*args)# Dispatch generically
>>>...define or import func3...>>>args = (2,3)>>>args += (4,)>>>args(2, 3, 4) >>>func3(*args)
def tracer(func, *pargs, **kargs):# Accept arbitrary argumentsprint('calling:', func.__name__) return func(*pargs, **kargs)# Pass along arbitrary argumentsdef func(a, b, c, d): return a + b + c + d print(tracer(func, 1, 2, c=3, d=4))
calling: func 10
func(*pargs, **kargs)# Newer call syntax: func(*sequence, **dict)apply(func, pargs, kargs)# Defunct built-in: apply(func, sequence, dict)
>>>def echo(*args, **kwargs): print(args, kwargs)>>>echo(1, 2, a=3, b=4)(1, 2) {'a': 3, 'b': 4}
>>>pargs = (1, 2)>>>kargs = {'a':3, 'b':4}>>>apply(echo, pargs, kargs)(1, 2) {'a': 3, 'b': 4} >>>echo(*pargs, **kargs)(1, 2) {'a': 3, 'b': 4}
>>>apply(pow, (2, 100))1267650600228229401496703205376L >>>pow(*(2, 100))1267650600228229401496703205376L
>>>echo(0, c=5, *pargs, **kargs)# Normal, keyword, *sequence, **dictionary(0, 1, 2) {'a': 3, 'c': 5, 'b': 4}
>>>def kwonly(a, *b, c):print(a, b, c)>>>kwonly(1, 2, c=3)1 (2,) 3 >>>kwonly(a=1, c=3)1 () 3 >>>kwonly(1, 2, 3)TypeError: kwonly() missing 1 required keyword-only argument: 'c'
>>>def kwonly(a, *, b, c):print(a, b, c)>>>kwonly(1, c=3, b=2)1 2 3 >>>kwonly(c=3, b=2, a=1)1 2 3 >>>kwonly(1, 2, 3)TypeError: kwonly() takes 1 positional argument but 3 were given >>>kwonly(1)TypeError: kwonly() missing 2 required keyword-only arguments: 'b' and 'c'
>>>def kwonly(a, *, b='spam', c='ham'):print(a, b, c)>>>kwonly(1)1 spam ham >>>kwonly(1, c=3)1 spam 3 >>>kwonly(a=1)1 spam ham >>>kwonly(c=3, b=2, a=1)1 2 3 >>>kwonly(1, 2)TypeError: kwonly() takes 1 positional argument but 2 were given
>>>def kwonly(a, *, b, c='spam'):print(a, b, c)>>>kwonly(1, b='eggs')1 eggs spam >>>kwonly(1, c='eggs')TypeError: kwonly() missing 1 required keyword-only argument: 'b' >>>kwonly(1, 2)TypeError: kwonly() takes 1 positional argument but 2 were given >>>def kwonly(a, *, b=1, c, d=2):print(a, b, c, d)>>>kwonly(3, c=4)3 1 4 2 >>>kwonly(3, c=4, b=5)3 5 4 2 >>>kwonly(3)TypeError: kwonly() missing 1 required keyword-only argument: 'c' >>>kwonly(1, 2, 3)TypeError: kwonly() takes 1 positional argument but 3 were given
>>>def kwonly(a, **pargs, b, c):SyntaxError: invalid syntax >>>def kwonly(a, **, b, c):SyntaxError: invalid syntax
>>>def f(a, *b, **d, c=6): print(a, b, c, d)# Keyword-only before **!SyntaxError: invalid syntax >>>def f(a, *b, c=6, **d): print(a, b, c, d)# Collect args in header>>>f(1, 2, 3, x=4, y=5)# Default used1 (2, 3) 6 {'y': 5, 'x': 4} >>>f(1, 2, 3, x=4, y=5, c=7)# Override default1 (2, 3) 7 {'y': 5, 'x': 4} >>>f(1, 2, 3, c=7, x=4, y=5)# Anywhere in keywords1 (2, 3) 7 {'y': 5, 'x': 4} >>>def f(a, c=6, *b, **d): print(a, b, c, d)# c is not keyword-only here!>>>f(1, 2, 3, x=4)1 (3,) 2 {'x': 4}
>>>def f(a, *b, c=6, **d): print(a, b, c, d)# KW-only between * and **>>>f(1, *(2, 3), **dict(x=4, y=5))# Unpack args at call1 (2, 3) 6 {'y': 5, 'x': 4} >>>f(1, *(2, 3), **dict(x=4, y=5), c=7)# Keywords before **args!SyntaxError: invalid syntax >>>f(1, *(2, 3), c=7, **dict(x=4, y=5))# Override default1 (2, 3) 7 {'y': 5, 'x': 4} >>>f(1, c=7, *(2, 3), **dict(x=4, y=5))# After or before *1 (2, 3) 7 {'y': 5, 'x': 4} >>>f(1, *(2, 3), **dict(x=4, y=5, c=7))# Keyword-only in **1 (2, 3) 7 {'y': 5, 'x': 4}
process(X, Y, Z)# Use flag's defaultprocess(X, Y, notify=True)# Override flag default
def process(*args, notify=False): ...
args is a tuple) and traverses the rest by
slicing off the first (there’s no point in comparing an object to
itself, especially if it might be a large structure).list call and employs the list
sort method.def min1(*args):
res = args[0]
for arg in args[1:]:
if arg < res:
res = arg
return res
def min2(first, *rest):
for arg in rest:
if arg < first:
first = arg
return first
def min3(*args):
tmp = list(args) # Or, in Python 2.4+: return sorted(args)[0]
tmp.sort()
return tmp[0]
print(min1(3, 4, 1, 2))
print(min2("bb", "aa"))
print(min3([2,2], [1,1], [3,3]))% python mins.py
1
aa
[1, 1]def minmax(test, *args):
res = args[0]
for arg in args[1:]:
if test(arg, res):
res = arg
return res
def lessthan(x, y): return x < y # See also: lambda, eval
def grtrthan(x, y): return x > y
print(minmax(lessthan, 4, 2, 1, 5, 6, 3)) # Self-test code
print(minmax(grtrthan, 4, 2, 1, 5, 6, 3))
% python minmax.py
1
6def intersect(*args):
res = []
for x in args[0]: # Scan first sequence
if x in res: continue # Skip duplicates
for other in args[1:]: # For all other args
if x not in other: break # Item in each one?
else: # No: break out of loop
res.append(x) # Yes: add items to end
return res
def union(*args):
res = []
for seq in args: # For all args
for x in seq: # For all nodes
if not x in res:
res.append(x) # Add new items to result
return res%python>>>from inter2 import intersect, union>>>s1, s2, s3 = "SPAM", "SCAM", "SLAM">>>intersect(s1, s2), union(s1, s2)# Two operands(['S', 'A', 'M'], ['S', 'P', 'A', 'M', 'C']) >>>intersect([1, 2, 3], (1, 4))# Mixed types[1] >>>intersect(s1, s2, s3)# Three operands['S', 'A', 'M'] >>>union(s1, s2, s3)['S', 'P', 'A', 'M', 'C', 'L']
>>>def tester(func, items, trace=True):for i in range(len(items)):items = items[1:] + items[:1]if trace: print(items)print(sorted(func(*items)))>>>tester(intersect, ('a', 'abcdefg', 'abdst', 'albmcnd'))('abcdefg', 'abdst', 'albmcnd', 'a') ['a'] ('abdst', 'albmcnd', 'a', 'abcdefg') ['a'] ('albmcnd', 'a', 'abcdefg', 'abdst') ['a'] ('a', 'abcdefg', 'abdst', 'albmcnd') ['a'] >>>tester(union, ('a', 'abcdefg', 'abdst', 'albmcnd'), False)['a', 'b', 'c', 'd', 'e', 'f', 'g', 'l', 'm', 'n', 's', 't'] ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'l', 'm', 'n', 's', 't'] ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'l', 'm', 'n', 's', 't'] ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'l', 'm', 'n', 's', 't'] >>>tester(intersect, ('ba', 'abcdefg', 'abdst', 'albmcnd'), False)['a', 'b'] ['a', 'b'] ['a', 'b'] ['a', 'b']
>>>intersect([1, 2, 1, 3], (1, 1, 4))[1] >>>union([1, 2, 1, 3], (1, 1, 4))[1, 2, 3, 4] >>>tester(intersect, ('ababa', 'abcdefga', 'aaaab'), False)['a', 'b'] ['a', 'b'] ['a', 'b']
>>>def tester(func, items, trace=True):for args in scramble(items):...use args...
from __future__ import print_function
#!python
"""
Emulate most of the 3.X print function for use in 2.X (and 3.X).
Call signature: print3(*args, sep=' ', end='\n', file=sys.stdout)
"""
import sys
def print3(*args, **kargs):
sep = kargs.get('sep', ' ') # Keyword arg defaults
end = kargs.get('end', '\n')
file = kargs.get('file', sys.stdout)
output = ''
first = True
for arg in args:
output += ('' if first else sep) + str(arg)
first = False
file.write(output + end)from print3 import print3 print3(1, 2, 3) print3(1, 2, 3, sep='')# Suppress separatorprint3(1, 2, 3, sep='...') print3(1, [2], (3,), sep='...')# Various object typesprint3(4, 5, 6, sep='', end='')# Suppress newlineprint3(7, 8, 9) print3()# Add newline (or blank line)import sys print3(1, 2, 3, sep='??', end='.\n', file=sys.stderr)# Redirect to file
C:\code> c:\python27\python testprint3.py
1 2 3
123
1...2...3
1...[2]...(3,)
4567 8 9
1??2??3.#!python3
"Use 3.X only keyword-only args"
import sys
def print3(*args, sep=' ', end='\n', file=sys.stdout):
output = ''
first = True
for arg in args:
output += ('' if first else sep) + str(arg)
first = False
file.write(output + end)>>> print3(99, name='bob')
TypeError: print3() got an unexpected keyword argument 'name'#!python
"Use 2.X/3.X keyword args deletion with defaults"
import sys
def print3(*args, **kargs):
sep = kargs.pop('sep', ' ')
end = kargs.pop('end', '\n')
file = kargs.pop('file', sys.stdout)
if kargs: raise TypeError('extra keywords: %s' % kargs)
output = ''
first = True
for arg in args:
output += ('' if first else sep) + str(arg)
first = False
file.write(output + end)>>> print3(99, name='bob')
TypeError: extra keywords: {'name': 'bob'}>>>def func(a, b=4, c=5):print(a, b, c)>>>func(1, 2)
>>>def func(a, b, c=5):print(a, b, c)>>>func(1, c=3, b=2)
>>>def func(a, *pargs):print(a, pargs)>>>func(1, 2, 3)
>>>def func(a, **kargs):print(a, kargs)>>>func(a=1, c=3, b=2)
>>>def func(a, b, c=3, d=4): print(a, b, c, d)>>>func(1, *(5, 6))
>>>def func(a, b, c): a = 2; b[0] = 'x'; c['a'] = 'y'>>>l=1; m=[1]; n={'a':0}>>>func(l, m, n)>>>l, m, n
1 2 5,
because 1 and 2 are passed to a and b
by position, and c is omitted in
the call and defaults to 5.1 2
3: 1 is passed to
a by position, and b and c
are passed 2 and 3 by name (the left-to-right order doesn’t
matter when keyword arguments are used like this).1 (2, 3),
because 1 is passed to a and the *pargs collects the remaining positional
arguments into a new tuple object. We can step through the extra
positional arguments tuple with any iteration tool (e.g., for arg in pargs: ...).1 {'b': 2,
'c': 3}, because 1 is
passed to a by name and the
**kargs collects the remaining
keyword arguments into a dictionary. We could step through the extra
keyword arguments dictionary by key with any iteration tool (e.g.,
for key in kargs: ...). Note that
the order of the dictionary’s keys may vary per Python and other
variables.1 5 6 4:
the 1 matches a by position, 5 and 6
match b and c by *name positionals (6 overrides c’s default), and d defaults to 4 because it was not passed a value.(1, ['x'], {'a':
'y'}) — the first assignment in the function doesn’t impact
the caller, but the second two do because they change passed-in
mutable objects in place.1 Actually, this is fairly complicated. The Python sort routine is coded in C and uses a
highly optimized algorithm that attempts to take advantage of
partial ordering in the items to be sorted. It’s named “timsort”
after Tim Peters, its creator, and in its documentation it claims to
have “supernatural performance” at times (pretty good, for a sort!).
Still, sorting is an inherently exponential operation (it must chop
up the sequence and put it back together many times), and the other
versions simply perform one linear left-to-right scan. The net
effect is that sorting is quicker if the arguments are partially
ordered, but is likely to be slower otherwise (this still holds true
in test runs in 3.3). Even so, Python performance can change over
time, and the fact that sorting is implemented in the C language can
help greatly; for an exact analysis, you should time the
alternatives with the time or
timeit modules — we’ll see how in
Chapter 21.
return for outputs. Generally, you should strive to
make a function independent of things outside of it. Arguments and
return statements are often the
best ways to isolate external dependencies to a small number of
well-known places in your code.>>>def mysum(L):if not L:return 0else:return L[0] + mysum(L[1:])# Call myself recursively>>>mysum([1, 2, 3, 4, 5])15
>>>def mysum(L):print(L)# Trace recursive levelsif not L:# L shorter at each levelreturn 0else:return L[0] + mysum(L[1:])>>>mysum([1, 2, 3, 4, 5])[1, 2, 3, 4, 5] [2, 3, 4, 5] [3, 4, 5] [4, 5] [5] [] 15
def mysum(L):
return 0 if not L else L[0] + mysum(L[1:]) # Use ternary expression
def mysum(L):
return L[0] if len(L) == 1 else L[0] + mysum(L[1:]) # Any type, assume one
def mysum(L):
first, *rest = L
return first if not rest else first + mysum(rest) # Use 3.X ext seq assign>>>mysum([1])# mysum([]) fails in last 21 >>>mysum([1, 2, 3, 4, 5])15 >>>mysum(('s', 'p', 'a', 'm'))# But various types now work'spam' >>>mysum(['spam', 'ham', 'eggs'])'spamhameggs'
mysum('spam')), because strings
are sequences of one-character strings.mysum(open(name))), but the others do not because they
index (Chapter 14 illustrates
extended sequence assignment on files).def mysum(first,
*rest), although similar to the third variant, wouldn’t
work at all, because it expects individual arguments, not a single
iterable.>>>def mysum(L):if not L: return 0return nonempty(L)# Call a function that calls me>>>def nonempty(L):return L[0] + mysum(L[1:])# Indirectly recursive>>>mysum([1.1, 2.2, 3.3, 4.4])11.0
>>>L = [1, 2, 3, 4, 5]>>>sum = 0>>>while L:sum += L[0]L = L[1:]>>>sum15
>>>L = [1, 2, 3, 4, 5]>>>sum = 0>>>for x in L: sum += x>>>sum15
[1, [2, [3, 4], 5], 6, [7, 8]] # Arbitrarily nested sublists# file sumtree.pydef sumtree(L): tot = 0 for x in L:# For each item at this levelif not isinstance(x, list): tot += x# Add numbers directlyelse: tot += sumtree(x)# Recur for sublistsreturn tot L = [1, [2, [3, 4], 5], 6, [7, 8]]# Arbitrary nestingprint(sumtree(L))# Prints 36# Pathological casesprint(sumtree([1, [2, [3, [4, [5]]]]]))# Prints 15 (right-heavy)print(sumtree([[[[[1], 2], 3], 4], 5]))# Prints 15 (left-heavy)
def sumtree(L):# Breadth-first, explicit queuetot = 0 items = list(L)# Start with copy of top levelwhile items: front = items.pop(0)# Fetch/delete front itemif not isinstance(front, list): tot += front# Add numbers directlyelse: items.extend(front)# <== Append all in nested listreturn tot
def sumtree(L):# Depth-first, explicit stacktot = 0 items = list(L)# Start with copy of top levelwhile items: front = items.pop(0)# Fetch/delete front itemif not isinstance(front, list): tot += front# Add numbers directlyelse: items[:0] = front# <== Prepend all in nested listreturn tot
c:\code> sumtree2.py
1, 6, 2, 5, 7, 8, 3, 4, 36
1, 2, 3, 4, 5, 15
5, 4, 3, 2, 1, 15
----------------------------------------
1, 2, 3, 4, 5, 6, 7, 8, 36
1, 2, 3, 4, 5, 15
1, 2, 3, 4, 5, 15
---------------------------------------- if state not in visited:
visited.add(state) # x.add(state), x[state]=True, or x.append(state)
...proceed... visited.add(front)
...proceed...
items.extend([x for x in front if x not in visited])>>>sys.getrecursionlimit()# 1000 calls deep default1000 >>>sys.setrecursionlimit(10000)# Allow deeper nesting>>>help(sys.setrecursionlimit)# Read more about it
>>>def echo(message):# Name echo assigned to function objectprint(message)>>>echo('Direct call')# Call object through original nameDirect call >>>x = echo# Now x references the function too>>>x('Indirect call!')# Call object through name by adding ()Indirect call!
>>>def indirect(func, arg):func(arg)# Call the passed-in object by adding ()>>>indirect(echo, 'Argument call!')# Pass the function to another functionArgument call!
>>>schedule = [ (echo, 'Spam!'), (echo, 'Ham!') ]>>>for (func, arg) in schedule:func(arg)# Call functions embedded in containersSpam! Ham!
>>>def make(label):# Make a function but don't call itdef echo(message):print(label + ':' + message)return echo>>>F = make('Spam')# Label in enclosing scope is retained>>>F('Ham!')# Call the function that make returnedSpam:Ham! >>>F('Eggs!')Spam:Eggs!
>>>def func(a):b = 'spam'return b * a>>>func(8)'spamspamspamspamspamspamspamspam'
>>>func.__name__'func' >>>dir(func)['__annotations__', '__call__', '__class__', '__closure__', '__code__',...more omitted: 34 total...'__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>>func.__code__<code object func at 0x00000000021A6030, file "<stdin>", line 1> >>>dir(func.__code__)['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__',...more omitted: 37 total...'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames'] >>>func.__code__.co_varnames('a', 'b') >>>func.__code__.co_argcount1
>>>func<function func at 0x000000000296A1E0> >>>func.count = 0>>>func.count += 1>>>func.count1 >>>func.handles = 'Button-Press'>>>func.handles'Button-Press' >>>dir(func)['__annotations__', '__call__', '__class__', '__closure__', '__code__',...and more: in 3.X all others have double underscores so your names won't clash...__str__', '__subclasshook__', 'count', 'handles']
c:\code>py −3>>>def f(): pass>>>dir(f)...run on your own to see...>>>len(dir(f))34 >>>[x for x in dir(f) if not x.startswith('__')][] c:\code>py −2>>>def f(): pass>>>dir(f)...run on your own to see...>>>len(dir(f))31 >>>[x for x in dir(f) if not x.startswith('__')]['func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']
>>>def func(a, b, c):return a + b + c>>>func(1, 2, 3)6
>>>def func(a: 'spam', b: (1, 10), c: float) -> int:return a + b + c>>>func(1, 2, 3)6
>>> func.__annotations__
{'c': <class 'float'>, 'b': (1, 10), 'a': 'spam', 'return': <class 'int'>}>>>def func(a: 'spam', b, c: 99):return a + b + c>>>func(1, 2, 3)6 >>>func.__annotations__{'c': 99, 'a': 'spam'} >>>for arg in func.__annotations__:print(arg, '=>', func.__annotations__[arg])c => 99 a => spam
>>>def func(a: 'spam' = 4, b: (1, 10) = 5, c: float = 6) -> int:return a + b + c>>>func(1, 2, 3)6 >>>func()# 4 + 5 + 6 (all defaults)15 >>>func(1, c=10)# 1 + 5 + 10 (keywords work normally)16 >>>func.__annotations__{'c': <class 'float'>, 'b': (1, 10), 'a': 'spam', 'return': <class 'int'>}
>>>def func(a:'spam'=4, b:(1,10)=5, c:float=6)->int:return a + b + c>>>func(1, 2)# 1 + 2 + 69 >>>func.__annotations__{'c': <class 'float'>, 'b': (1, 10), 'a': 'spam', 'return': <class 'int'>}
lambdaargument1,argument2,...argumentN:expression using arguments
lambda is an expression, not a statement. Because
of this, a lambda can appear in
places a def is not allowed by
Python’s syntax — inside a list literal or a function call’s
arguments, for example. With def,
functions can be referenced by name but must be created elsewhere.
As an expression, lambda returns
a value (a new function) that can optionally be assigned a name. In
contrast, the def statement
always assigns the new function to the name in the header, instead
of returning it as a result.lambda’s body is a single expression, not a block of
statements. The lambda’s body is similar to what you’d put
in a def body’s return statement; you simply type the
result as a naked expression, instead of explicitly returning it.
Because it is limited to an expression, a lambda is less general than a def — you can only squeeze so much logic
into a lambda body without using
statements such as if. This is by
design, to limit program nesting: lambda is designed for coding simple
functions, and def handles larger
tasks.>>>def func(x, y, z): return x + y + z>>>func(2, 3, 4)9
>>>f = lambda x, y, z: x + y + z>>>f(2, 3, 4)9
>>>x = (lambda a="fee", b="fie", c="foe": a + b + c)>>>x("wee")'weefiefoe'
>>>def knights():title = 'Sir'action = (lambda x: title + ' ' + x)# Title in enclosing def scopereturn action# Return a function object>>>act = knights()>>>msg =act('robin')# 'robin' passed to x>>>msg'Sir robin' >>>act# act: a function, not its result<function knights.<locals>.<lambda> at 0x00000000029CA488>
L = [lambda x: x ** 2,# Inline function definitionlambda x: x ** 3, lambda x: x ** 4]# A list of three callable functionsfor f in L: print(f(2))# Prints 4, 8, 16print(L[0](3))# Prints 9
def f1(x): return x ** 2 def f2(x): return x ** 3# Define named functionsdef f3(x): return x ** 4 L = [f1, f2, f3]# Reference by namefor f in L: print(f(2))# Prints 4, 8, 16print(L[0](3))# Prints 9
>>>key = 'got'>>>{'already': (lambda: 2 + 2),'got': (lambda: 2 * 4),'one': (lambda: 2 ** 6)}[key]()8
>>>def f1(): return 2 + 2>>>def f2(): return 2 * 4>>>def f3(): return 2 ** 6>>>key = 'one'>>>{'already': f1, 'got': f2, 'one': f3}[key]()64
if a:
b
else:
cb if a else c ((a and b) or c)
>>>lower = (lambda x, y: x if x < y else y)>>>lower('bb', 'aa')'aa' >>>lower('aa', 'bb')'aa'
>>>import sys>>>showall = lambda x: list(map(sys.stdout.write, x))# 3.X: must use list>>>t = showall(['spam\n', 'toast\n', 'eggs\n'])# 3.X: can use printspam toast eggs >>>showall = lambda x: [sys.stdout.write(line) for line in x]>>>t = showall(('bright\n', 'side\n', 'of\n', 'life\n'))bright side of life >>>showall = lambda x: [print(line, end='') for line in x]# Same: 3.X only>>>showall = lambda x: print(*x, sep='', end='')# Same: 3.X only
>>>def action(x):return (lambda y: x + y)# Make and return function, remember x>>>act = action(99)>>>act<function action.<locals>.<lambda> at 0x00000000029CA2F0> >>>act(2)# Call what action returned101
>>>action = (lambda x: (lambda y: x + y))>>>act = action(99)>>>act(3)102 >>>((lambda x: (lambda y: x + y))(99))(4)103
>>>counters = [1, 2, 3, 4]>>> >>>updated = []>>>for x in counters:updated.append(x + 10)# Add 10 to each item>>>updated[11, 12, 13, 14]
>>>def inc(x): return x + 10# Function to be run>>>list(map(inc, counters))# Collect results[11, 12, 13, 14]
>>>list(map((lambda x: x + 3), counters))# Function expression[4, 5, 6, 7]
>>>def mymap(func, seq):res = []for x in seq: res.append(func(x))return res
>>>list(map(inc, [1, 2, 3]))# Built-in is an iterable[11, 12, 13] >>>mymap(inc, [1, 2, 3])# Ours builds a list (see generators)[11, 12, 13]
>>>pow(3, 4)# 3**481 >>>list(map(pow, [1, 2, 3], [2, 3, 4]))# 1**2, 2**3, 3**4[1, 8, 81]
>>>list(map(inc, [1, 2, 3, 4]))[11, 12, 13, 14] >>>[inc(x) for x in [1, 2, 3, 4]]# Use () parens to generate items instead[11, 12, 13, 14]
>>>list(range(−5, 5))# An iterable in 3.X[−5, −4, −3, −2, −1, 0, 1, 2, 3, 4] >>>list(filter((lambda x: x > 0), range(−5, 5)))# An iterable in 3.X[1, 2, 3, 4]
>>>res = []>>>for x in range(−5, 5):# The statement equivalentif x > 0:res.append(x)>>>res[1, 2, 3, 4]
>>>[x for x in range(−5, 5) if x > 0]# Use () to generate items[1, 2, 3, 4]
>>>from functools import reduce# Import in 3.X, not in 2.X>>>reduce((lambda x, y: x + y), [1, 2, 3, 4])10 >>>reduce((lambda x, y: x * y), [1, 2, 3, 4])24
>>>L = [1,2,3,4]>>>res = L[0]>>>for x in L[1:]:res = res + x>>> res 10
>>>def myreduce(function, sequence):tally = sequence[0]for next in sequence[1:]:tally = function(tally, next)return tally>>>myreduce((lambda x, y: x + y), [1, 2, 3, 4, 5])15 >>>myreduce((lambda x, y: x * y), [1, 2, 3, 4, 5])120
>>>import operator, functools>>>functools.reduce(operator.add, [2, 4, 6])# Function-based +12 >>>functools.reduce((lambda x, y: x + y), [2, 4, 6])12
lambda
expressions and def statements
related?lambda?map,
filter, and reduce.lambda and def create function objects to be called
later. Because lambda is an
expression, though, it returns a function object instead of assigning
it to a name, and it can be used to nest a function definition in
places where a def will not work
syntactically. A lambda allows for
only a single implicit return value expression, though; because it
does not support a block of statements, it is not ideal for larger
functions.lambdas allow us to “inline”
small units of executable code, defer its execution, and provide it
with state in the form of default arguments and enclosing scope
variables. Using a lambda is never
required; you can always code a def
instead and reference the function by name. lambdas come in handy, though, to embed
small pieces of deferred code that are unlikely to be used elsewhere
in a program. They commonly appear in callback-based programs such as
GUIs, and they have a natural affinity with functional tools like
map and filter that expect a processing
function.map passes each item to the
function and collects all results, filter collects items for which the function
returns a True value, and reduce computes a single value by applying
the function to an accumulator and successive items. Unlike the other
two, reduce is available in the
functools module in 3.X, not the
built-in scope; reduce is a
built-in in 2.X.__annotations__ attribute. Python places no
semantic meaning on these annotations, but simply packages them for
potential use by other tools.return statements, by changing passed-in
mutable arguments, and by setting global variables. Globals are
generally frowned upon (except for very special cases, like
multithreaded programs) because they can make code more difficult to
understand and use. return
statements are usually best, but changing mutables is fine (and even
useful), if expected. Functions may also communicate results with
system devices such as files and sockets, but these are beyond our
scope here.1 The lambda tends to
intimidate people more than it should. This reaction seems to stem
from the name “lambda” itself — a name that comes from the Lisp
language, which got it from lambda calculus, which is a form of
symbolic logic. In Python, though, it’s really just a keyword that
introduces the expression syntactically. Obscure mathematical heritage
aside, lambda is simpler to use
than you may think: it’s simply an alternative way to code a function,
albeit without full statements, decorators, or 3.X annotations.
2 A student once noted that you could skip the dispatch table
dictionary in such code if the function name is the same as its
string lookup key — run an eval(funcname)() to kick off the call. While true in
this case and sometimes useful, as we saw earlier (e.g., Chapter 10), eval is relatively slow (it must compile
and run code), and insecure (you must trust the string’s source).
More fundamentally, jump tables are generally subsumed by
polymorphic method dispatch in Python: calling a method does the
“right thing” based on the type of object. To see why, stay tuned
for Part VI.
>>> ord('s')
115>>>res = []>>>for x in 'spam':res.append(ord(x))# Manual results collection>>>res[115, 112, 97, 109]
>>>res = list(map(ord, 'spam'))# Apply function to sequence (or other)>>>res[115, 112, 97, 109]
>>>res = [ord(x) for x in 'spam']# Apply expression to sequence (or other)>>>res[115, 112, 97, 109]
>>> [x ** 2 for x in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]>>> list(map((lambda x: x ** 2), range(10)))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]>>>[x for x in range(5) if x % 2 == 0][0, 2, 4] >>>list(filter((lambda x: x % 2 == 0), range(5)))[0, 2, 4] >>>res = []>>>for x in range(5):if x % 2 == 0:res.append(x)>>>res[0, 2, 4]
>>> [x ** 2 for x in range(10) if x % 2 == 0]
[0, 4, 16, 36, 64]>>>list(map((lambda x: x**2), filter((lambda x: x % 2 == 0), range(10))) )[0, 4, 16, 36, 64]
[expressionfortargetiniterable]
[expressionfortarget1initerable1ifcondition1fortarget2initerable2ifcondition2... fortargetNiniterableNifconditionN]
>>>res = [x + y for x in [0, 1, 2] for y in [100, 200, 300]]>>>res[100, 200, 300, 101, 201, 301, 102, 202, 302]
>>>res = []>>>for x in [0, 1, 2]:for y in [100, 200, 300]:res.append(x + y)>>>res[100, 200, 300, 101, 201, 301, 102, 202, 302]
>>> [x + y for x in 'spam' for y in 'SPAM']
['sS', 'sP', 'sA', 'sM', 'pS', 'pP', 'pA', 'pM',
'aS', 'aP', 'aA', 'aM', 'mS', 'mP', 'mA', 'mM']>>>[x + y for x in 'spam' if x in 'sm' for y in 'SPAM' if y in ('P', 'A')]['sP', 'sA', 'mP', 'mA'] >>>[x + y + z for x in 'spam' if x in 'sm'for y in 'SPAM' if y in ('P', 'A')for z in '123' if z > '1']['sP2', 'sP3', 'sA2', 'sA3', 'mP2', 'mP3', 'mA2', 'mA3']
>>> [(x, y) for x in range(5) if x % 2 == 0 for y in range(5) if y % 2 == 1]
[(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]>>>res = []>>>for x in range(5):if x % 2 == 0:for y in range(5):if y % 2 == 1:res.append((x, y))>>>res[(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]
>>>M = [[1, 2, 3],[4, 5, 6],[7, 8, 9]]>>>N = [[2, 2, 2],[3, 3, 3],[4, 4, 4]]
>>>M[1]# Row 2[4, 5, 6] >>>M[1][2]# Row 2, item 36
>>>[row[1] for row in M]# Column 2[2, 5, 8] >>>[M[row][1] for row in (0, 1, 2)]# Using offsets[2, 5, 8]
>>>[M[i][i] for i in range(len(M))]# Diagonals[1, 5, 9] >>>[M[i][len(M)-1-i] for i in range(len(M))][3, 5, 7]
>>>L = [[1, 2, 3], [4, 5, 6]]>>>for i in range(len(L)):for j in range(len(L[i])):# Update in placeL[i][j] += 10>>>L[[11, 12, 13], [14, 15, 16]]
>>>[col + 10 for row in M for col in row]# Assign to M to retain new value[11, 12, 13, 14, 15, 16, 17, 18, 19] >>>[[col + 10 for col in row] for row in M][[11, 12, 13], [14, 15, 16], [17, 18, 19]]
>>>res = []>>>for row in M:# Statement equivalentsfor col in row:# Indent parts further rightres.append(col + 10)>>>res[11, 12, 13, 14, 15, 16, 17, 18, 19] >>>res = []>>>for row in M:tmp = []# Left-nesting starts new listfor col in row:tmp.append(col + 10)res.append(tmp)>>>res[[11, 12, 13], [14, 15, 16], [17, 18, 19]]
>>>M[[1, 2, 3], [4, 5, 6], [7, 8, 9]] >>>N[[2, 2, 2], [3, 3, 3], [4, 4, 4]] >>>[M[row][col] * N[row][col] for row in range(3) for col in range(3)][2, 4, 6, 12, 15, 18, 28, 32, 36] >>>[[M[row][col] * N[row][col] for col in range(3)] for row in range(3)][[2, 4, 6], [12, 15, 18], [28, 32, 36]]
res = []
for row in range(3):
tmp = []
for col in range(3):
tmp.append(M[row][col] * N[row][col])
res.append(tmp)[[col1 * col2 for (col1, col2) in zip(row1, row2)] for (row1, row2) in zip(M, N)]
res = []
for (row1, row2) in zip(M, N):
tmp = []
for (col1, col2) in zip(row1, row2):
tmp.append(col1 * col2)
res.append(tmp)Simple is better than complex.
def statements,
but use yield statements to return results one at a time, suspending and resuming
their state between each.>>>def gensquares(N):for i in range(N):yield i ** 2# Resume here later
>>>for i in gensquares(5):# Resume the functionprint(i, end=' : ')# Print last yielded value0 : 1 : 4 : 9 : 16 : >>>
>>>x = gensquares(4)>>>x<generator object gensquares at 0x000000000292CA68>
>>>next(x)# Same as x.__next__() in 3.X0 >>>next(x)# Use x.next() or next() in 2.X1 >>>next(x)4 >>>next(x)9 >>>next(x)Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
>>>y = gensquares(5)# Returns a generator which is its own iterator>>>iter(y) is y# iter() is not required: a no-op hereTrue >>>next(y)# Can run next()immediately0
>>>def buildsquares(n):res = []for i in range(n): res.append(i ** 2)return res>>>for x in buildsquares(5): print(x, end=' : ')0 : 1 : 4 : 9 : 16 :
>>>for x in [n ** 2 for n in range(5)]:print(x, end=' : ')0 : 1 : 4 : 9 : 16 : >>>for x in map((lambda n: n ** 2), range(5)):print(x, end=' : ')0 : 1 : 4 : 9 : 16 :
>>>def ups(line):for sub in line.split(','):# Substring generatoryield sub.upper()>>>tuple(ups('aaa,bbb,ccc'))# All iteration contexts('AAA', 'BBB', 'CCC') >>>{i: s for (i, s) in enumerate(ups('aaa,bbb,ccc'))}{0: 'AAA', 1: 'BBB', 2: 'CCC'}
>>>def gen():for i in range(10):X = yield iprint(X)>>>G = gen()>>>next(G)# Must call next() first, to start generator0 >>>G.send(77)# Advance, and send value to yield expression77 1 >>>G.send(88)88 2 >>>next(G)# next() and X.__next__() send NoneNone 3
>>>[x ** 2 for x in range(4)]# List comprehension: build a list[0, 1, 4, 9] >>>(x ** 2 for x in range(4))# Generator expression: make an iterable<generator object <genexpr> at 0x00000000029A8288>
>>>list(x ** 2 for x in range(4))# List comprehension equivalence[0, 1, 4, 9]
>>>G = (x ** 2 for x in range(4))>>>iter(G) is G# iter(G) optional: __iter__ returns selfTrue >>>next(G)# Generator objects: automatic methods0>>>next(G)1 >>>next(G)4 >>>next(G)9 >>>next(G)Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration >>>G<generator object <genexpr> at 0x00000000029A8318>
>>>for num in (x ** 2 for x in range(4)):# Calls next() automaticallyprint('%s, %s' % (num, num / 2.0))0, 0.0 1, 0.5 4, 2.0 9, 4.5
>>>''.join(x.upper() for x in 'aaa,bbb,ccc'.split(','))'AAABBBCCC' >>>a, b, c = (x + '\n' for x in 'aaa,bbb,ccc'.split(','))>>>a, c('aaa\n', 'ccc\n')
>>>sum(x ** 2 for x in range(4))# Parens optional14 >>>sorted(x ** 2 for x in range(4))# Parens optional[0, 1, 4, 9] >>>sorted((x ** 2 for x in range(4)), reverse=True)# Parens required[9, 4, 1, 0]
>>>list(map(abs, (−1, −2, 3, 4)))# Map function on tuple[1, 2, 3, 4] >>>list(abs(x) for x in (−1, −2, 3, 4))# Generator expression[1, 2, 3, 4] >>>list(map(lambda x: x * 2, (1, 2, 3, 4)))# Nonfunction case[2, 4, 6, 8] >>>list(x * 2 for x in (1, 2, 3, 4))# Simpler as generator?[2, 4, 6, 8]
>>>line = 'aaa,bbb,ccc'>>>''.join([x.upper() for x in line.split(',')])# Makes a pointless list'AAABBBCCC' >>>''.join(x.upper() for x in line.split(','))# Generates results'AAABBBCCC' >>>''.join(map(str.upper, line.split(',')))# Generates results'AAABBBCCC' >>>''.join(x * 2 for x in line.split(','))# Simpler as generator?'aaaaaabbbbbbcccccc' >>>''.join(map(lambda x: x * 2, line.split(',')))'aaaaaabbbbbbcccccc'
>>>[x * 2 for x in [abs(x) for x in (−1, −2, 3, 4)]]# Nested comprehensions[2, 4, 6, 8] >>>list(map(lambda x: x * 2, map(abs, (−1, −2, 3, 4))))# Nested maps[2, 4, 6, 8] >>>list(x * 2 for x in (abs(x) for x in (−1, −2, 3, 4)))# Nested generators[2, 4, 6, 8]
>>>import math>>>list(map(math.sqrt, (x ** 2 for x in range(4))))# Nested combinations[0.0, 1.0, 2.0, 3.0]
>>>list(map(abs, map(abs, map(abs, (−1, 0, 1)))))# Nesting gone bad?[1, 0, 1] >>>list(abs(x) for x in (abs(x) for x in (abs(x) for x in (−1, 0, 1))))[1, 0, 1]
>>>list(abs(x) * 2 for x in (−1, −2, 3, 4))# Unnested equivalents[2, 4, 6, 8] >>>list(math.sqrt(x ** 2) for x in range(4))# Flat is often better[0.0, 1.0, 2.0, 3.0] >>>list(abs(x) for x in (−1, 0, 1))[1, 0, 1]
>>>line = 'aa bbb c'>>>''.join(x for x in line.split() if len(x) > 1)# Generator with 'if''aabbb' >>>''.join(filter(lambda x: len(x) > 1, line.split()))# Similar to filter'aabbb'
>>>''.join(x.upper() for x in line.split() if len(x) > 1)'AABBB' >>>''.join(map(str.upper, filter(lambda x: len(x) > 1, line.split())))'AABBB'
>>>''.join(x.upper() for x in line.split() if len(x) > 1)'AABBB' >>>res = ''>>>for x in line.split():# Statement equivalent?if len(x) > 1:# This is also a joinres += x.upper()>>>res'AABBB'
A functiondefstatement that contains ayieldstatement is turned into a generator function. When called, it returns a new generator object with automatic retention of local scope and code position; an automatically created__iter__method that simply returns itself; and an automatically created__next__method (nextin 2.X) that starts the function or resumes it where it last left off, and raisesStopIterationwhen finished producing results.
A comprehension expression enclosed in parentheses is known as a generator expression. When run, it returns a new generator object with the same automatically created method interface and state retention as a generator function call’s results — with an__iter__method that simply returns itself; and a_next__method (nextin 2.X) that starts the implied loop or resumes it where it last left off, and raisesStopIterationwhen finished producing results.
>>>G = (c * 4 for c in 'SPAM')# Generator expression>>>list(G)# Force generator to produce all results['SSSS', 'PPPP', 'AAAA', 'MMMM']
>>>def timesfour(S):# Generator functionfor c in S:yield c * 4>>>G = timesfour('spam')>>>list(G)# Iterate automatically['ssss', 'pppp', 'aaaa', 'mmmm']
>>>G = (c * 4 for c in 'SPAM') >>>I = iter(G)# Iterate manually (expression)>>>next(I)'SSSS' >>>next(I)'PPPP' >>>G = timesfour('spam')>>>I = iter(G)# Iterate manually (function)>>>next(I)'ssss' >>>next(I)'pppp'
>>>line = 'aa bbb c'>>>''.join(x.upper() for x in line.split() if len(x) > 1)# Expression'AABBB' >>>def gensub(line):# Functionfor x in line.split():if len(x) > 1:yield x.upper()>>>''.join(gensub(line))# But why generate?'AABBB'
>>>G = (c * 4 for c in 'SPAM')>>>iter(G) is G# My iterator is myself: G has __next__True
>>>G = (c * 4 for c in 'SPAM')# Make a new generator>>>I1 = iter(G)# Iterate manually>>>next(I1)'SSSS' >>>next(I1)'PPPP' >>>I2 = iter(G)# Second iterator at same position!>>>next(I2)'AAAA'
>>>list(I1)# Collect the rest of I1's items['MMMM'] >>>next(I2)# Other iterators exhausted tooStopIteration >>>I3 = iter(G)# Ditto for new iterators>>>next(I3)StopIteration >>>I3 = iter(c * 4 for c in 'SPAM')# New generator to start over>>>next(I3)'SSSS'
>>>def timesfour(S):for c in S:yield c * 4>>>G = timesfour('spam')# Generator functions work the same way>>>iter(G) is GTrue >>>I1, I2 = iter(G), iter(G)>>>next(I1)'ssss' >>>next(I1)'pppp' >>>next(I2)# I2 at same position as I1'aaaa'
>>>L = [1, 2, 3, 4]>>>I1, I2 = iter(L), iter(L)>>>next(I1)1 >>>next(I1)2 >>>next(I2)# Lists support multiple iterators1 >>>del L[2:]# Changes reflected in iterators>>>next(I1)StopIteration
>>>D = {'a':1, 'b':2, 'c':3}>>>x = iter(D)>>>next(x)'c' >>>next(x)'b'
>>>for key in D:print(key, D[key])c 3 b 2 a 1
>>>for line in open('temp.txt'):print(line, end='')Tis but a flesh wound.
>>>import os>>>for (root, subs, files) in os.walk('.'):# Directory walk generatorfor name in files:# A Python 'find' operationif name.startswith('call'):print(root, name). callables.py .\dualpkg callables.py
>>>G = os.walk(r'C:\code\pkg')>>>iter(G) is G# Single-scan iterator: iter(G) optionalTrue >>>I = iter(G)>>>next(I)('C:\\code\\pkg', ['__pycache__'], ['eggs.py', 'eggs.pyc', 'main.py',...etc...]) >>>next(I)('C:\\code\\pkg\\__pycache__', [], ['eggs.cpython-33.pyc',...etc...]) >>>next(I)StopIteration
>>>def f(a, b, c): print('%s, %s, and %s' % (a, b, c))>>>f(0, 1, 2)# Normal positionals0, 1, and 2 >>>f(*range(3))# Unpack range values: iterable in 3.X0, 1, and 2 >>>f(*(i for i in range(3)))# Unpack generator expression values0, 1, and 2
>>>D = dict(a='Bob', b='dev', c=40.5); D{'b': 'dev', 'c': 40.5, 'a': 'Bob'} >>>f(a='Bob', b='dev', c=40.5)# Normal keywordsBob, dev, and 40.5 >>>f(**D)# Unpack dict: key=valueBob, dev, and 40.5 >>>f(*D)# Unpack keys iteratorb, c, and a >>>f(*D.values())# Unpack view iterator: iterable in 3.Xdev, 40.5, and Bob
>>>for x in 'spam': print(x.upper(), end=' ')S P A M >>>list(print(x.upper(), end=' ') for x in 'spam')S P A M [None, None, None, None] >>>print(*(x.upper() for x in 'spam'))S P A M
class SomeIterable:
def __iter__(...): ... # On iter(): return self or supplemental object
def __next__(...): ... # On next(): coded here, or in another class>>>L, S = [1, 2, 3], 'spam'>>>for i in range(len(S)):# For repeat counts 0..3S = S[1:] + S[:1]# Move front item to the endprint(S, end=' ')pams amsp mspa spam >>>for i in range(len(L)):L = L[1:] + L[:1]# Slice so any sequence type worksprint(L, end=' ')[2, 3, 1] [3, 1, 2] [1, 2, 3]
>>>for i in range(len(S)):# For positions 0..3X = S[i:] + S[:i]# Rear part + front part (same effect)print(X, end=' ')spam pams amsp mspa
>>>def scramble(seq):res = []for i in range(len(seq)):res.append(seq[i:] + seq[:i])return res>>>scramble('spam')['spam', 'pams', 'amsp', 'mspa'] >>>def scramble(seq):return [seq[i:] + seq[:i] for i in range(len(seq))]>>>scramble('spam')['spam', 'pams', 'amsp', 'mspa'] >>>for x in scramble((1, 2, 3)):print(x, end=' ')(1, 2, 3) (2, 3, 1) (3, 1, 2)
>>>def scramble(seq):for i in range(len(seq)):seq = seq[1:] + seq[:1]# Generator functionyield seq# Assignments work here>>>def scramble(seq):for i in range(len(seq)):# Generator functionyield seq[i:] + seq[:i]# Yield one item per iteration>>>list(scramble('spam'))# list() generates all results['spam', 'pams', 'amsp', 'mspa'] >>>list(scramble((1, 2, 3)))# Any sequence type works[(1, 2, 3), (2, 3, 1), (3, 1, 2)] >>> >>>for x in scramble((1, 2, 3)):# for loops generate resultsprint(x, end=' ')(1, 2, 3) (2, 3, 1) (3, 1, 2)
>>>S'spam' >>>G = (S[i:] + S[:i] for i in range(len(S)))# Generator expression equivalent>>>list(G)['spam', 'pams', 'amsp', 'mspa']
>>>F = lambda seq: (seq[i:] + seq[:i] for i in range(len(seq)))>>>F(S)<generator object <genexpr> at 0x00000000029883F0> >>> >>>list(F(S))['spam', 'pams', 'amsp', 'mspa'] >>>list(F([1, 2, 3]))[[1, 2, 3], [2, 3, 1], [3, 1, 2]] >>>for x in F((1, 2, 3)):print(x, end=' ')(1, 2, 3) (2, 3, 1) (3, 1, 2)
# file scramble.pydef scramble(seq): for i in range(len(seq)):# Generator functionyield seq[i:] + seq[:i]# Yield one item per iterationscramble2 = lambda seq: (seq[i:] + seq[:i] for i in range(len(seq)))
>>>from scramble import scramble>>>from inter2 import intersect, union>>> >>>def tester(func, items, trace=True):for args in scramble(items):# Use generator (or: scramble2(items))if trace: print(args)print(sorted(func(*args)))>>>tester(intersect, ('aab', 'abcde', 'ababab'))('aab', 'abcde', 'ababab') ['a', 'b'] ('abcde', 'ababab', 'aab') ['a', 'b'] ('ababab', 'aab', 'abcde') ['a', 'b'] >>>tester(intersect, ([1, 2], [2, 3, 4], [1, 6, 2, 7, 3]), False)[2] [2] [2]
# File permute.pydef permute1(seq): if not seq:# Shuffle any sequence: listreturn [seq]# Empty sequenceelse: res = [] for i in range(len(seq)): rest = seq[:i] + seq[i+1:]# Delete current nodefor x in permute1(rest):# Permute the othersres.append(seq[i:i+1] + x)# Add node at frontreturn res def permute2(seq): if not seq:# Shuffle any sequence: generatoryield seq# Empty sequenceelse: for i in range(len(seq)): rest = seq[:i] + seq[i+1:]# Delete current nodefor x in permute2(rest):# Permute the othersyield seq[i:i+1] + x# Add node at front
>>>from scramble import scramble>>>from permute import permute1, permute2>>>list(scramble('abc'))# Simple scrambles: N['abc', 'bca', 'cab'] >>>permute1('abc')# Permutations larger: N!['abc', 'acb', 'bac', 'bca', 'cab', 'cba'] >>>list(permute2('abc'))# Generate all combinations['abc', 'acb', 'bac', 'bca', 'cab', 'cba'] >>>G = permute2('abc')# Iterate (iter() not needed)>>>next(G)'abc' >>>next(G)'acb' >>>for x in permute2('abc'): print(x)# Automatic iteration...prints six lines...
>>>permute1('spam') == list(permute2('spam'))True >>>len(list(permute2('spam'))), len(list(scramble('spam')))(24, 4) >>>list(scramble('spam'))['spam', 'pams', 'amsp', 'mspa'] >>>list(permute2('spam'))['spam', 'spma', 'sapm', 'samp', 'smpa', 'smap', 'psam', 'psma', 'pasm', 'pams', 'pmsa', 'pmas', 'aspm', 'asmp', 'apsm', 'apms', 'amsp', 'amps', 'mspa', 'msap', 'mpsa', 'mpas', 'masp', 'maps']
Explicit is better than implicit.
>>>import math>>>math.factorial(10)# 10 * 9 * 8 * 7 * 6 * 5 * 4 * 3 * 2 * 13628800 >>>from permute import permute1, permute2>>>seq = list(range(10))>>>p1 = permute1(seq)# 37 seconds on a 2GHz quad-core machine# Creates a list of 3.6M numbers>>>len(p1), p1[0], p1[1](3628800, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 9, 8])
>>>p2 = permute2(seq)# Returns generator immediately>>>next(p2)# And produces each result quickly on request[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>>next(p2)[0, 1, 2, 3, 4, 5, 6, 7, 9, 8] >>>p2 = list(permute2(seq))# About 28 seconds, though still impractical>>>p1 == p2# Same set of results generatedTrue
>>>math.factorial(50)30414093201713378043612608166064768844377641568960512000000000000 >>>p3 = permute2(list(range(50)))>>>next(p3)# permute1 is not an option here![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
>>>import random>>>math.factorial(20)# permute1 is not an option here2432902008176640000 >>>seq = list(range(20))>>>random.shuffle(seq)# Shuffle sequence randomly first>>>p = permute2(seq)>>>next(p)[10, 17, 4, 14, 11, 3, 16, 19, 12, 8, 6, 5, 2, 15, 18, 7, 1, 0, 13, 9] >>>next(p)[10, 17, 4, 14, 11, 3, 16, 19, 12, 8, 6, 5, 2, 15, 18, 7, 1, 0, 9, 13] >>>random.shuffle(seq)>>>p = permute2(seq)>>>next(p)[16, 1, 5, 14, 15, 12, 0, 2, 6, 19, 10, 17, 11, 18, 13, 7, 4, 9, 8, 3] >>>next(p)[16, 1, 5, 14, 15, 12, 0, 2, 6, 19, 10, 17, 11, 18, 13, 7, 4, 9, 3, 8]
>>>S1 = 'abc'>>>S2 = 'xyz123'>>>list(zip(S1, S2))# zip pairs items from iterables[('a', 'x'), ('b', 'y'), ('c', 'z')]# zip pairs items, truncates at shortest>>>list(zip([−2, −1, 0, 1, 2]))# Single sequence: 1-ary tuples[(−2,), (−1,), (0,), (1,), (2,)] >>>list(zip([1, 2, 3], [2, 3, 4, 5]))# N sequences: N-ary tuples[(1, 2), (2, 3), (3, 4)]# map passes paired items to function, truncates>>>list(map(abs, [−2, −1, 0, 1, 2]))# Single sequence: 1-ary function[2, 1, 0, 1, 2] >>>list(map(pow, [1, 2, 3], [2, 3, 4, 5]))# N sequences: N-ary function, 3.X[1, 8, 81]# map and zip accept arbitrary iterables>>>list(map(lambda x, y: x + y, open('script2.py'), open('script2.py')))['import sys\nimport sys\n', 'print(sys.path)\nprint(sys.path)\n',...etc...] >>>[x + y for (x, y) in zip(open('script2.py'), open('script2.py'))]['import sys\nimport sys\n', 'print(sys.path)\nprint(sys.path)\n',...etc...]
# map(func, seqs...) workalike with zip
def mymap(func, *seqs):
res = []
for args in zip(*seqs):
res.append(func(*args))
return res
print(mymap(abs, [-2, −1, 0, 1, 2]))
print(mymap(pow, [1, 2, 3], [2, 3, 4, 5]))[2, 1, 0, 1, 2] [1, 8, 81]
# Using a list comprehension
def mymap(func, *seqs):
return [func(*args) for args in zip(*seqs)]
print(mymap(abs, [−2, −1, 0, 1, 2]))
print(mymap(pow, [1, 2, 3], [2, 3, 4, 5]))# Using generators: yield and (...)
def mymap(func, *seqs):
for args in zip(*seqs):
yield func(*args)
def mymap(func, *seqs):
return (func(*args) for args in zip(*seqs))print(list(mymap(abs, [−2, −1, 0, 1, 2]))) print(list(mymap(pow, [1, 2, 3], [2, 3, 4, 5])))
C:code>c:\python27\python>>>map(None, [1, 2, 3], [2, 3, 4, 5])[(1, 2), (2, 3), (3, 4), (None, 5)] >>>map(None, 'abc', 'xyz123')[('a', 'x'), ('b', 'y'), ('c', 'z'), (None, '1'), (None, '2'), (None, '3')]
# zip(seqs...) and 2.X map(None, seqs...) workalikes
def myzip(*seqs):
seqs = [list(S) for S in seqs]
res = []
while all(seqs):
res.append(tuple(S.pop(0) for S in seqs))
return res
def mymapPad(*seqs, pad=None):
seqs = [list(S) for S in seqs]
res = []
while any(seqs):
res.append(tuple((S.pop(0) if S else pad) for S in seqs))
return res
S1, S2 = 'abc', 'xyz123'
print(myzip(S1, S2))
print(mymapPad(S1, S2))
print(mymapPad(S1, S2, pad=99))[('a', 'x'), ('b', 'y'), ('c', 'z')]
[('a', 'x'), ('b', 'y'), ('c', 'z'), (None, '1'), (None, '2'), (None, '3')]
[('a', 'x'), ('b', 'y'), ('c', 'z'), (99, '1'), (99, '2'), (99, '3')]# Using generators: yield
def myzip(*seqs):
seqs = [list(S) for S in seqs]
while all(seqs):
yield tuple(S.pop(0) for S in seqs)
def mymapPad(*seqs, pad=None):
seqs = [list(S) for S in seqs]
while any(seqs):
yield tuple((S.pop(0) if S else pad) for S in seqs)
S1, S2 = 'abc', 'xyz123'
print(list(myzip(S1, S2)))
print(list(mymapPad(S1, S2)))
print(list(mymapPad(S1, S2, pad=99)))# Alternate implementation with lengths
def myzip(*seqs):
minlen = min(len(S) for S in seqs)
return [tuple(S[i] for S in seqs) for i in range(minlen)]
def mymapPad(*seqs, pad=None):
maxlen = max(len(S) for S in seqs)
index = range(maxlen)
return [tuple((S[i] if len(S) > i else pad) for S in seqs) for i in index]
S1, S2 = 'abc', 'xyz123'
print(myzip(S1, S2))
print(mymapPad(S1, S2))
print(mymapPad(S1, S2, pad=99))# Using generators: (...)def myzip(*seqs): minlen = min(len(S) for S in seqs) return (tuple(S[i] for S in seqs) for i in range(minlen)) S1, S2 = 'abc', 'xyz123' print(list(myzip(S1, S2)))# Go!... [('a', 'x'), ('b', 'y'), ('c', 'z')]
{1, 3, 2} is equivalent to set([1, 3, 2]), and the new set
comprehension syntax {f(x) for x in S if
P(x)} is like the generator expression set(f(x) for x in S if P(x)), where f(x) is an arbitrary expression.{key: val for (key, val)
in zip(keys, vals)} works like the form dict(zip(keys, vals)), and {x: f(x) for x in items} is like the
generator expression dict((x, f(x)) for x in
items).>>>[x * x for x in range(10)]# List comprehension: builds list[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]# Like list(generator expr)>>>(x * x for x in range(10))# Generator expression: produces items<generator object at 0x009E7328># Parens are often optional>>>{x * x for x in range(10)}# Set comprehension, 3.X and 2.7{0, 1, 4, 81, 64, 9, 16, 49, 25, 36}# {x, y} is a set in these versions too>>>{x: x * x for x in range(10)}# Dictionary comprehension, 3.X and 2.7{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}
c:\code>py −3>>>(X for X in range(5))<generator object <genexpr> at 0x00000000028E4798> >>>XNameError: name 'X' is not defined >>>X = 99>>>[X for X in range(5)]# 3.X: generator, set, dict, and list localize[0, 1, 2, 3, 4] >>>X99 >>>Y = 99>>>for Y in range(5): pass# But loop statements do not localize names>>>Y4
>>>X = 'aaa'>>>def func():Y = 'bbb'print(''.join(Z for Z in X + Y))# Z comprehension, Y local, X global>>>func()aaabbb
c:\code>py −2>>>(X for X in range(5))<generator object <genexpr> at 0x0000000002147EE8> >>>XNameError: name 'X' is not defined >>>X = 99>>>[X for X in range(5)]# 2.X: List does not localize its names, like for[0, 1, 2, 3, 4] >>>X4 >>>Y = 99>>>for Y in range(5): pass# for loops do not localize names in 2.X or 3.X>>>Y4
>>>{x * x for x in range(10)}# Comprehension{0, 1, 4, 81, 64, 9, 16, 49, 25, 36} >>>set(x * x for x in range(10))# Generator and type name{0, 1, 4, 81, 64, 9, 16, 49, 25, 36} >>>{x: x * x for x in range(10)}{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81} >>>dict((x, x * x) for x in range(10)){0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81} >>>x# Loop variable localized in 2.X + 3.XNameError: name 'x' is not defined
>>>res = set()>>>for x in range(10):# Set comprehension equivalentres.add(x * x)>>>res{0, 1, 4, 81, 64, 9, 16, 49, 25, 36} >>>res = {}>>>for x in range(10):# Dict comprehension equivalentres[x] = x * x>>>res{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81} >>>x# Localized in comprehension expressions, but not in loop statements9
>>>G = ((x, x * x) for x in range(10))>>>next(G)(0, 0) >>>next(G)(1, 1)
>>>[x * x for x in range(10) if x % 2 == 0]# Lists are ordered[0, 4, 16, 36, 64] >>>{x * x for x in range(10) if x % 2 == 0}# But sets are not{0, 16, 4, 64, 36} >>>{x: x * x for x in range(10) if x % 2 == 0}# Neither are dict keys{0: 0, 8: 64, 2: 4, 4: 16, 6: 36}
>>>[x + y for x in [1, 2, 3] for y in [4, 5, 6]]# Lists keep duplicates[5, 6, 7, 6, 7, 8, 7, 8, 9] >>>{x + y for x in [1, 2, 3] for y in [4, 5, 6]}# But sets do not{8, 9, 5, 6, 7} >>>{x: y for x in [1, 2, 3] for y in [4, 5, 6]}# Neither do dict keys{1: 6, 2: 6, 3: 6}
>>>{x + y for x in 'ab' for y in 'cd'}{'ac', 'bd', 'bc', 'ad'} >>>{x + y: (ord(x), ord(y)) for x in 'ab' for y in 'cd'}{'ac': (97, 99), 'bd': (98, 100), 'bc': (98, 99), 'ad': (97, 100)} >>>{k * 2 for k in ['spam', 'ham', 'sausage'] if k[0] == 's'}{'sausagesausage', 'spamspam'} >>>{k.upper(): k * 2 for k in ['spam', 'ham', 'sausage'] if k[0] == 's'}{'SAUSAGE': 'sausagesausage', 'SPAM': 'spamspam'}
yield statement
do?map calls and list
comprehensions related? Compare and contrast the two.__next__ method (next in 2.X) that repeatedly advances to the
next item in a series of results and raises an exception at the end of
the series. In Python, we can code generator functions with def and yield, generator expressions with
parenthesized comprehensions, and generator objects with classes that
define a special method named __iter__ (discussed later in the
book).yield statement somewhere in its code.
Generator functions are otherwise identical to normal functions
syntactically, but they are compiled specially by Python so as to
return an iterable generator object when called. That object retains
state and code location between values.yield statement is run, it sends a
result back to the caller and suspends the function’s state; the
function can then be resumed after the last yield statement, in response to a next built-in or __next__ method call issued by the caller.
In more advanced roles, the generator send method similarly resumes the generator,
but can also pass a value that shows up as the yield expression’s value. Generator
functions may also have a return
statement, which terminates the generator.map call is similar to a
list comprehension — both produce a series of values, by collecting the
results of applying an operation to each item in a sequence or other
iterable, one item at a time. The primary difference is that map applies a function call to each item,
and list comprehensions apply arbitrary expressions. Because of this,
list comprehensions are more general; they can apply a function call
expression like map, but map requires a function to apply other kinds
of expressions. List comprehensions also support extended syntax such
as nested for loops and if clauses that subsume the filter built-in. In Python 3.X, map also differs in that it produces a
generator of values; the list comprehension
materializes the result list in memory all at once. In 2.X, both tools
create result lists.1 Technically, Python treats return statement values in
generator functions as syntax errors in 2.X, and in all 3.X prior
to 3.3. As of 3.3, a return statement value is allowed and
attached to the StopIteration object, but the
value is ignored in automatic iterations contexts, and using this
makes code incompatible with all prior releases.
2 Interestingly, generator functions are also something of a
“poor man’s” multithreading device — they
interleave a function’s work with that of its caller, by dividing
its operation into steps run between yields. Generators are not threads,
though: the program is explicitly directed to and from the
function within a single thread of control. In one sense,
threading is more general (producers can run truly independently
and post results to a queue), but generators may be simpler to
code. See the footnote in Chapter 17 for a brief
introduction to Python multithreading tools. Note that because
control is routed explicitly at yield and next calls, generators are also not
backtracking, but are more strongly related
to coroutines — formal concepts that are both
beyond this chapter’s scope.
# File timer0.pyimport time def timer(func, *args):# Simplistic timing functionstart = time.clock() for i in range(1000): func(*args) return time.clock() - start# Total elapsed time in seconds
>>>from timer0 import timer>>>timer(pow, 2, 1000)# Time to call pow(2, 1000) 1000 times0.00296260674205626 >>>timer(str.upper, 'spam')# Time to call 'spam'.upper() 1000 times0.0005165746166859719
range
to the tested function’s timetime.clock,
which might not be best outside Windows# File timer.py""" Homegrown timing tools for function calls. Does total time, best-of time, and best-of-totals time """ import time, sys timer = time.clock if sys.platform[:3] == 'win' else time.time def total(reps, func, *pargs, **kargs): """ Total time to run func() reps times. Returns (total time, last result) """ repslist = list(range(reps))# Hoist out, equalize 2.x, 3.xstart = timer()# Or perf_counter/other in 3.3+for i in repslist: ret = func(*pargs, **kargs) elapsed = timer() - start return (elapsed, ret) def bestof(reps, func, *pargs, **kargs): """ Quickest func() among reps runs. Returns (best time, last result) """ best = 2 ** 32# 136 years seems large enoughfor i in range(reps):# range usage not timed herestart = timer() ret = func(*pargs, **kargs) elapsed = timer() - start# Or call total() with reps=1if elapsed < best: best = elapsed# Or add to list and take min()return (best, ret) def bestoftotal(reps1, reps2, func, *pargs, **kargs): """ Best of totals: (best of reps1 runs of (total of reps2 runs of func)) """ return bestof(reps1, total, reps2, func, *pargs, **kargs)
time module gives
access to the current time, with precision that varies per platform.
On Windows its clock function
is claimed to give microsecond granularity and so is very accurate.
Because the time function
may be better on Unix, this script selects between them
automatically based on the platform string in the sys module; it starts with “win” if running in Windows. See also
the sidebar “New Timer Calls in 3.3” on other
time options in 3.3 and later not
used here for portability; we will also be timing Python 2.X where
these newer calls are not available, and their results on Windows
appear similar in 3.3 in any event.range call is hoisted
out of the timing loop in the total function, so its construction cost
is not charged to the timed function in Python 2.X. In 3.X range is an iterable, so this step is
neither required nor harmful, but we still run the result through
list so its traversal cost is the
same in both 2.X and 3.X. This doesn’t apply to the bestof function, since no range factors are charged to the test’s
time.reps count is passed in
as an argument, before the test function and its arguments, to allow
repetition to vary per call.bestoftotal function at the end. See Chapter 18 for a refresher if this code doesn’t make
sense.>>>import timer>>>timer.total(1000, pow, 2, 1000)[0]# Compare to timer0 results above0.0029542985410557776 >>>timer.total(1000, str.upper, 'spam')# Returns (time, last call's result)(0.000504845391709686, 'SPAM') >>>timer.bestof(1000, str.upper, 'spam')# 1/1000 as long as total time(4.887177027512735e-07, 'SPAM') >>>timer.bestof(1000, pow, 2, 1000000)[0]0.00393515497972885 >>>timer.bestof(50, timer.total, 1000, str.upper, 'spam')(0.0005468751145372153, (0.0005004469323637295, 'SPAM')) >>>timer.bestoftotal(50, 1000, str.upper, 'spam')(0.000566912540591602, (0.0005195069228989269, 'SPAM'))
>>> min(timer.total(1000, str.upper, 'spam') for i in range(50))
(0.0005155971812769167, 'SPAM')>>>((((2 ** 32) / 60) / 60) / 24) / 365# Plus a few extra days136.19251953323186 >>>((((2 ** 32) // 60) // 60) // 24) // 365# Floor: see Chapter 5136
# File timeseqs.py"Test the relative speed of iteration tool alternatives." import sys, timer# Import timer functionsreps = 10000 repslist = list(range(reps))# Hoist out, list in both 2.X/3.Xdef forLoop(): res = [] for x in repslist: res.append(abs(x)) return res def listComp(): return [abs(x) for x in repslist] def mapCall(): return list(map(abs, repslist))# Use list() here in 3.X only!# return map(abs, repslist) def genExpr(): return list(abs(x) for x in repslist)# list() required to force resultsdef genFunc(): def gen(): for x in repslist: yield abs(x) return list(gen())# list() required to force resultsprint(sys.version) for test in (forLoop, listComp, mapCall, genExpr, genFunc): (bestof, (total, result)) = timer.bestoftotal(5, 1000, test) print ('%-9s: %.5f => [%s...%s]' % (test.__name__, bestof, result[0], result[-1]))
C:\code> c:\python33\python timeseqs.py
3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
forLoop : 1.33290 => [0...9999]
listComp : 0.69658 => [0...9999]
mapCall : 0.56483 => [0...9999]
genExpr : 1.08457 => [0...9999]
genFunc : 1.07623 => [0...9999]return [abs(x) for x in repslist]# 0.69 secondsreturn list(abs(x) for x in repslist)# 1.08 seconds: differs internally
c:\code> c:\python27\python timeseqs.py
2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)]
forLoop : 1.24902 => [0...9999]
listComp : 0.66970 => [0...9999]
mapCall : 0.57018 => [0...9999]
genExpr : 0.90339 => [0...9999]
genFunc : 0.90542 => [0...9999]c:\code> c:\PyPy\pypy-1.9\pypy.exe timeseqs.py
2.7.2 (341e1e3821ff, Jun 07 2012, 15:43:00)
[PyPy 1.9.0 with MSC v.1500 32 bit]
forLoop : 0.10106 => [0...9999]
listComp : 0.05629 => [0...9999]
mapCall : 0.10022 => [0...9999]
genExpr : 0.17234 => [0...9999]
genFunc : 0.17519 => [0...9999]# File timeseqs2.py (differing parts)... def forLoop(): res = [] for x in repslist: res.append(x + 10) return res def listComp(): return [x + 10 for x in repslist] def mapCall(): return list(map((lambda x: x + 10), repslist))# list() in 3.X onlydef genExpr(): return list(x + 10 for x in repslist)# list() in 2.X + 3.Xdef genFunc(): def gen(): for x in repslist: yield x + 10 return list(gen())# list in 2.X + 3.X...
c:\code> c:\python33\python timeseqs2.py
3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
forLoop : 1.35136 => [10...10009]
listComp : 0.73730 => [10...10009]
mapCall : 1.68588 => [10...10009]
genExpr : 1.10963 => [10...10009]
genFunc : 1.11074 => [10...10009]def F(x): return x
def listComp():
return [F(x) for x in repslist]
def mapCall():
return list(map(F, repslist))# File timer2.py (2.X and 3.X)""" total(spam, 1, 2, a=3, b=4, _reps=1000) calls and times spam(1, 2, a=3, b=4) _reps times, and returns total time for all runs, with final result. bestof(spam, 1, 2, a=3, b=4, _reps=5) runs best-of-N timer to attempt to filter out system load variation, and returns best time among _reps tests. bestoftotal(spam, 1, 2, a=3, b=4, _reps1=5, _reps=1000) runs best-of-totals test, which takes the best among _reps1 runs of (the total of _reps runs); """ import time, sys timer = time.clock if sys.platform[:3] == 'win' else time.time def total(func, *pargs, **kargs): _reps = kargs.pop('_reps', 1000)# Passed-in or default repsrepslist = list(range(_reps))# Hoist range out for 2.X listsstart = timer() for i in repslist: ret = func(*pargs, **kargs) elapsed = timer() - start return (elapsed, ret) def bestof(func, *pargs, **kargs): _reps = kargs.pop('_reps', 5) best = 2 ** 32 for i in range(_reps): start = timer() ret = func(*pargs, **kargs) elapsed = timer() - start if elapsed < best: best = elapsed return (best, ret) def bestoftotal(func, *pargs, **kargs): _reps1 = kargs.pop('_reps1', 5) return min(total(func, *pargs, **kargs) for i in range(_reps1))
import sys, timer2
...
for test in (forLoop, listComp, mapCall, genExpr, genFunc):
(total, result) = timer2.bestoftotal(test, _reps1=5, _reps=1000)
# Or:
# (total, result) = timer2.bestoftotal(test)
# (total, result) = timer2.bestof(test, _reps=5)
# (total, result) = timer2.total(test, _reps=1000)
# (bestof, (total, result)) = timer2.bestof(timer2.total, test, _reps=5)
print ('%-9s: %.5f => [%s...%s]' %
(test.__name__, total, result[0], result[-1]))>>>from timer2 import total, bestof, bestoftotal>>>total(pow, 2, 1000)[0]# 2 ** 1000, 1K dflt reps0.0029562534118596773 >>>total(pow, 2, 1000, _reps=1000)[0]# 2 ** 1000, 1K reps0.0029733585316193967 >>>total(pow, 2, 1000, _reps=1000000)[0]# 2 ** 1000, 1M reps1.2451676814889865 >>>bestof(pow, 2, 100000)[0]# 2 ** 100K, 5 dflt reps0.0007550688578703557 >>>bestof(pow, 2, 1000000, _reps=30)[0]# 2 ** 1M, best of 300.004040229286800923 >>>bestoftotal(str.upper, 'spam', _reps1=30, _reps=1000)# Best of 30, tot of 1K(0.0004945823198454491, 'SPAM') >>>bestof(total, str.upper, 'spam', _reps=30)# Nested calls work too(0.0005463863968202531, (0.0004994694969298052, 'SPAM'))
>>>def spam(a, b, c, d): return a + b + c + d>>>total(spam, 1, 2, c=3, d=4, _reps=1000)(0.0009730369554290519, 10) >>>bestof(spam, 1, 2, c=3, d=4, _reps=1000)(9.774353202374186e-07, 10) >>>bestoftotal(spam, 1, 2, c=3, d=4, _reps1=1000, _reps=1000)(0.00037289161070930277, 10) >>>bestoftotal(spam, *(1, 2), _reps1=1000, _reps=1000, **dict(c=3, d=4))(0.00037289161070930277, 10)
# File timer3.py (3.X only)
"""
Same usage as timer2.py, but uses 3.X keyword-only default arguments
instead of dict pops for simpler code. No need to hoist range() out
of tests in 3.X: always a generator in 3.X, and this can't run on 2.X.
"""
import time, sys
timer = time.clock if sys.platform[:3] == 'win' else time.time
def total(func, *pargs, _reps=1000, **kargs):
start = timer()
for i in range(_reps):
ret = func(*pargs, **kargs)
elapsed = timer() - start
return (elapsed, ret)
def bestof(func, *pargs, _reps=5, **kargs):
best = 2 ** 32
for i in range(_reps):
start = timer()
ret = func(*pargs, **kargs)
elapsed = timer() - start
if elapsed < best: best = elapsed
return (best, ret)
def bestoftotal(func, *pargs, _reps1=5, **kargs):
return min(total(func, *pargs, **kargs) for i in range(_reps1))(elapsed, ret) = total(func, *pargs, _reps=1, **kargs)
c:\code>py −3Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit... >>>import timeit>>>min(timeit.repeat(stmt="[x ** 2 for x in range(1000)]", number=1000, repeat=5))0.5062382371756811 c:\code>py −2Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] on win32 >>>import timeit>>>min(timeit.repeat(stmt="[x ** 2 for x in range(1000)]", number=1000, repeat=5))0.0708020004193198 c:\code>c:\pypy\pypy-1.9\pypy.exePython 2.7.2 (341e1e3821ff, Jun 07 2012, 15:43:00) [PyPy 1.9.0 with MSC v.1500 32 bit] on win32 >>>>import timeit>>>>min(timeit.repeat(stmt="[x ** 2 for x in range(1000)]", number=1000, repeat=5))0.0059330329674303905
c:\code>C:\python33\Lib\timeit.py -n 1000 "[x ** 2 for x in range(1000)]"1000 loops, best of 3: 506 usec per loop c:\code>python -m timeit -n 1000 "[x ** 2 for x in range(1000)]"1000 loops, best of 3: 504 usec per loop c:\code>py −3 -m timeit -n 1000 -r 5 "[x ** 2 for x in range(1000)]"1000 loops, best of 5: 505 usec per loop
c:\code>set PATH=%PATH%;C:\pypy\pypy-1.9c:\code>py −3 -m timeit -n 1000 -r 5 -c "[x ** 2 for x in range(1000)]"1000 loops, best of 5: 502 usec per loop c:\code>py −2 -m timeit -n 1000 -r 5 -c "[x ** 2 for x in range(1000)]"1000 loops, best of 5: 70.6 usec per loop c:\code>pypy -m timeit -n 1000 -r 5 -c "[x ** 2 for x in range(1000)]"1000 loops, best of 5: 5.44 usec per loop C:\code>py −3 -m timeit -n 1000 -r 5 -c "[abs(x) for x in range(10000)]"1000 loops, best of 5: 815 usec per loop C:\code>py −2 -m timeit -n 1000 -r 5 -c "[abs(x) for x in range(10000)]"1000 loops, best of 5: 700 usec per loop C:\code>pypy -m timeit -n 1000 -r 5 -c "[abs(x) for x in range(10000)]"1000 loops, best of 5: 61.7 usec per loop
c:\code>py −3>>>import timeit>>>min(timeit.repeat(number=10000, repeat=3,stmt="L = [1, 2, 3, 4, 5]\nfor i in range(len(L)): L[i] += 1"))0.01397292797131814 >>>min(timeit.repeat(number=10000, repeat=3,stmt="L = [1, 2, 3, 4, 5]\ni=0\nwhile i < len(L):\n\tL[i] += 1\n\ti += 1"))0.015452276471516813 >>>min(timeit.repeat(number=10000, repeat=3,stmt="L = [1, 2, 3, 4, 5]\nM = [x + 1 for x in L]"))0.009464995838568635
c:\code>py −3 -m timeit -n 1000 -r 3 "L = [1,2,3,4,5]" "i=0" "while i < len(L):"" L[i] += 1" " i += 1"1000 loops, best of 3: 1.54 usec per loop c:\code>py −3 -m timeit -n 1000 -r 3 "L = [1,2,3,4,5]" "M = [x + 1 for x in L]"1000 loops, best of 3: 0.959 usec per loop
c:\code>python -m timeit -n 1000 -r 3 "L = [1,2,3,4,5]" "M = [x + 1 for x in L]"1000 loops, best of 3: 0.956 usec per loop c:\code>python -m timeit -n 1000 -r 3 -s "L = [1,2,3,4,5]" "M = [x + 1 for x in L]"1000 loops, best of 3: 0.775 usec per loop
>>>from timeit import repeat>>>min(repeat(number=1000, repeat=3,setup='from mins import min1, min2, min3\n''vals=list(range(1000))',stmt= 'min3(*vals)'))0.0387865921275079 >>>min(repeat(number=1000, repeat=3,setup='from mins import min1, min2, min3\n''import random\nvals=[random.random() for i in range(1000)]',stmt= 'min3(*vals)'))0.275656482278373
c:\code>py −3>>>import timeit>>>timeit.timeit(stmt='[x ** 2 for x in range(1000)]', number=1000)# Total time0.5238125259325834 >>>timeit.Timer(stmt='[x ** 2 for x in range(1000)]').timeit(1000)# Class API0.5282652329644009 >>>timeit.repeat(stmt='[x ** 2 for x in range(1000)]', number=1000, repeat=3)[0.5299034147194845, 0.5082454007998365, 0.5095136232504416] >>>def testcase():y = [x ** 2 for x in range(1000)]# Callable objects or code strings>>>min(timeit.repeat(stmt=testcase, number=1000, repeat=3))0.5073828140463377
"""pybench.py: Test speed of one or more Pythons on a set of simple code-string benchmarks. A function, to allow stmts to vary. This system itself runs on both 2.X and 3.X, and may spawn both. Uses timeit to test either the Python running this script by API calls, or a set of Pythons by reading spawned command-line outputs (os.popen) with Python's -m flag to find timeit on module search path. Replaces $listif3 with a list() around generators for 3.X and an empty string for 2.X, so 3.X does same work as 2.X. In command-line mode only, must split multiline statements into one separate quoted argument per line so all will be run (else might run/time first line only), and replace all \t in indentation with 4 spaces for uniformity. Caveats: command-line mode (only) may fail if test stmt embeds double quotes, quoted stmt string is incompatible with shell in general, or command-line exceeds a length limit on platform's shell--use API call mode or homegrown timer; does not yet support a setup statement: as is, time of all statements in the test stmt are charged to the total time. """ import sys, os, timeit defnum, defrep= 1000, 5# May vary per stmtdef runner(stmts, pythons=None, tracecmd=False): """ Main logic: run tests per input lists, caller handles usage modes. stmts: [(number?, repeat?, stmt-string)], replaces $listif3 in stmt pythons: None=this python only, or [(ispy3?, python-executable-path)] """ print(sys.version) for (number, repeat, stmt) in stmts: number = number or defnum repeat = repeat or defrep# 0=defaultif not pythons:# Run stmt on this python: API call# No need to split lines or quote hereispy3 = sys.version[0] == '3' stmt = stmt.replace('$listif3', 'list' if ispy3 else '') best = min(timeit.repeat(stmt=stmt, number=number, repeat=repeat)) print('%.4f [%r]' % (best, stmt[:70])) else:# Run stmt on all pythons: command line# Split lines into quoted argumentsprint('-' * 80) print('[%r]' % stmt) for (ispy3, python) in pythons: stmt1 = stmt.replace('$listif3', 'list' if ispy3 else '') stmt1 = stmt1.replace('\t', ' ' * 4) lines = stmt1.split('\n') args = ' '.join('"%s"' % line for line in lines) cmd = '%s -m timeit -n %s -r %s %s' % (python, number, repeat, args) print(python) if tracecmd: print(cmd) print('\t' + os.popen(cmd).read().rstrip())
"""pybench_cases.py: Run pybench on a set of pythons and statements. Select modes by editing this script or using command-line arguments (in sys.argv): e.g., run a "C:\python27\python pybench_cases.py" to test just one specific version on stmts, "pybench_cases.py -a" to test all pythons listed, or a "py −3 pybench_cases.py -a -t" to trace command lines too. """ import pybench, sys pythons = [# (ispy3?, path)(1, 'C:\python33\python'), (0, 'C:\python27\python'), (0, 'C:\pypy\pypy-1.9\pypy') ] stmts = [# (num,rpt,stmt)(0, 0, "[x ** 2 for x in range(1000)]"),# Iterations(0, 0, "res=[]\nfor x in range(1000): res.append(x ** 2)"),# \n=multistmt(0, 0, "$listif3(map(lambda x: x ** 2, range(1000)))"),# \n\t=indent(0, 0, "list(x ** 2 for x in range(1000))"),# $=list or ''(0, 0, "s = 'spam' * 2500\nx = [s[i] for i in range(10000)]"),# String ops(0, 0, "s = '?'\nfor i in range(10000): s += '?'"), ] tracecmd = '-t' in sys.argv# -t: trace command lines?pythons = pythons if '-a' in sys.argv else None# -a: all in list, else one?pybench.runner(stmts, pythons, tracecmd)
c:\code>py −3 pybench_cases.py3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] 0.5015 ['[x ** 2 for x in range(1000)]'] 0.5655 ['res=[]\nfor x in range(1000): res.append(x ** 2)'] 0.6044 ['list(map(lambda x: x ** 2, range(1000)))'] 0.5425 ['list(x ** 2 for x in range(1000))'] 0.8746 ["s = 'spam' * 2500\nx = [s[i] for i in range(10000)]"] 2.8060 ["s = '?'\nfor i in range(10000): s += '?'"] c:\code>py −2 pybench_cases.py2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] 0.0696 ['[x ** 2 for x in range(1000)]'] 0.1285 ['res=[]\nfor x in range(1000): res.append(x ** 2)'] 0.1636 ['(map(lambda x: x ** 2, range(1000)))'] 0.0952 ['list(x ** 2 for x in range(1000))'] 0.6143 ["s = 'spam' * 2500\nx = [s[i] for i in range(10000)]"] 2.0657 ["s = '?'\nfor i in range(10000): s += '?'"] c:\code>c:\pypy\pypy-1.9\pypy pybench_cases.py2.7.2 (341e1e3821ff, Jun 07 2012, 15:43:00) [PyPy 1.9.0 with MSC v.1500 32 bit] 0.0059 ['[x ** 2 for x in range(1000)]'] 0.0102 ['res=[]\nfor x in range(1000): res.append(x ** 2)'] 0.0099 ['(map(lambda x: x ** 2, range(1000)))'] 0.0156 ['list(x ** 2 for x in range(1000))'] 0.1298 ["s = 'spam' * 2500\nx = [s[i] for i in range(10000)]"] 5.5242 ["s = '?'\nfor i in range(10000): s += '?'"]
c:\code> py −3 pybench_cases.py -a
3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
--------------------------------------------------------------------------------
['[x ** 2 for x in range(1000)]']
C:\python33\python
1000 loops, best of 5: 499 usec per loop
C:\python27\python
1000 loops, best of 5: 71.4 usec per loop
C:\pypy\pypy-1.9\pypy
1000 loops, best of 5: 5.71 usec per loop
--------------------------------------------------------------------------------
['res=[]\nfor x in range(1000): res.append(x ** 2)']
C:\python33\python
1000 loops, best of 5: 562 usec per loop
C:\python27\python
1000 loops, best of 5: 130 usec per loop
C:\pypy\pypy-1.9\pypy
1000 loops, best of 5: 9.81 usec per loop
--------------------------------------------------------------------------------
['$listif3(map(lambda x: x ** 2, range(1000)))']
C:\python33\python
1000 loops, best of 5: 599 usec per loop
C:\python27\python
1000 loops, best of 5: 161 usec per loop
C:\pypy\pypy-1.9\pypy
1000 loops, best of 5: 9.45 usec per loop
--------------------------------------------------------------------------------
['list(x ** 2 for x in range(1000))']
C:\python33\python
1000 loops, best of 5: 540 usec per loop
C:\python27\python
1000 loops, best of 5: 92.3 usec per loop
C:\pypy\pypy-1.9\pypy
1000 loops, best of 5: 15.1 usec per loop
--------------------------------------------------------------------------------
["s = 'spam' * 2500\nx = [s[i] for i in range(10000)]"]
C:\python33\python
1000 loops, best of 5: 873 usec per loop
C:\python27\python
1000 loops, best of 5: 614 usec per loop
C:\pypy\pypy-1.9\pypy
1000 loops, best of 5: 118 usec per loop
--------------------------------------------------------------------------------
["s = '?'\nfor i in range(10000): s += '?'"]
C:\python33\python
1000 loops, best of 5: 2.81 msec per loop
C:\python27\python
1000 loops, best of 5: 1.94 msec per loop
C:\pypy\pypy-1.9\pypy
1000 loops, best of 5: 5.68 msec per looptimeit may skew results in
ways beyond our scope to explore here (e.g., garbage
collection).# pybench_cases2.pypythons += [ (1, 'C:\python32\python'), (0, 'C:\pypy\pypy-2.0-beta1\pypy')] stmts += [# Use function calls: map wins(0, 0, "[ord(x) for x in 'spam' * 2500]"), (0, 0, "res=[]\nfor x in 'spam' * 2500: res.append(ord(x))"), (0, 0, "$listif3(map(ord, 'spam' * 2500))"), (0, 0, "list(ord(x) for x in 'spam' * 2500)"),# Set and dicts(0, 0, "{x ** 2 for x in range(1000)}"), (0, 0, "s=set()\nfor x in range(1000): s.add(x ** 2)"), (0, 0, "{x: x ** 2 for x in range(1000)}"), (0, 0, "d={}\nfor x in range(1000): d[x] = x ** 2"),# Pathological: 300k digits(1, 1, "len(str(2**1000000))")]# Pypy loses on this today
c:\code> py −3 pybench_cases2.py
3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
0.7237 ["[ord(x) for x in 'spam' * 2500]"]
1.3471 ["res=[]\nfor x in 'spam' * 2500: res.append(ord(x))"]
0.6160 ["list(map(ord, 'spam' * 2500))"]
1.1244 ["list(ord(x) for x in 'spam' * 2500)"]
0.5446 ['{x ** 2 for x in range(1000)}']
0.6053 ['s=set()\nfor x in range(1000): s.add(x ** 2)']
0.5278 ['{x: x ** 2 for x in range(1000)}']
0.5414 ['d={}\nfor x in range(1000): d[x] = x ** 2']
1.8933 ['len(str(2**1000000))'] (0, 0, "f=open('C:/Python33/Lib/pdb.py')\nfor line in f: x=line\nf.close()"),c:\code>py −3 -m timeit -n 1000 -r 5 "f=open('C:/Python33/Lib/pdb.py')""for line in f: x=line" "f.close()">>>import timeit>>>min(timeit.repeat(number=1000, repeat=5,stmt="f=open('C:/Python33/Lib/pdb.py')\nfor line in f: x=line\nf.close()"))
stmts = [
(0, 0, "def f(x): return x\n[f(x) for x in 'spam' * 2500]"),
(0, 0, "def f(x): return x\nres=[]\nfor x in 'spam' * 2500: res.append(f(x))"),
(0, 0, "def f(x): return x\n$listif3(map(f, 'spam' * 2500))"),
(0, 0, "def f(x): return x\nlist(f(x) for x in 'spam' * 2500)")]
c:\code> py −3 pybench_cases2.py
3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
1.5400 ["def f(x): return x\n[f(x) for x in 'spam' * 2500]"]
2.0506 ["def f(x): return x\nres=[]\nfor x in 'spam' * 2500: res.append(f(x))"]
1.2489 ["def f(x): return x\nlist(map(f, 'spam' * 2500))"]
1.6526 ["def f(x): return x\nlist(f(x) for x in 'spam' * 2500)"]c:\code>py −3 timeseqs3.py3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] forLoop : 0.55022 => [0...998001] listComp : 0.48787 => [0...998001] mapCall : 0.59499 => [0...998001] genExpr : 0.52773 => [0...998001] genFunc : 0.52603 => [0...998001] c:\code>py −3 pybench_cases.py3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] 0.5015 ['[x ** 2 for x in range(1000)]'] 0.5657 ['res=[]\nfor x in range(1000): res.append(x ** 2)'] 0.6025 ['list(map(lambda x: x ** 2, range(1000)))'] 0.5404 ['list(x ** 2 for x in range(1000))'] 0.8711 ["s = 'spam' * 2500\nx = [s[i] for i in range(10000)]"] 2.8009 ["s = '?'\nfor i in range(10000): s += '?'"]
# pybench2.py... def runner(stmts, pythons=None, tracecmd=False): for (number, repeat,setup, stmt) in stmts: if not pythons: ... best = min(timeit.repeat(setup=setup, stmt=stmt, number=number, repeat=repeat)) else: setup = setup.replace('\t', ' ' * 4) setup = ' '.join('-s "%s"' % line for line in setup.split('\n')) ... for (ispy3, python) in pythons: ... cmd = '%s -m timeit -n %s -r %s%s%s' % (python, number, repeat,setup, args)# pybench2_cases.pyimport pybench2, sys ... stmts = [# (num,rpt,setup,stmt)(0, 0, "", "[x ** 2 for x in range(1000)]"), (0, 0, "", "res=[]\nfor x in range(1000): res.append(x ** 2)"), (0, 0, "def f(x):\n\treturn x", "[f(x) for x in 'spam' * 2500]"), (0, 0, "def f(x):\n\treturn x", "res=[]\nfor x in 'spam' * 2500:\n\tres.append(f(x))"), (0, 0, "L = [1, 2, 3, 4, 5]", "for i in range(len(L)): L[i] += 1"), (0, 0, "L = [1, 2, 3, 4, 5]", "i=0\nwhile i < len(L):\n\tL[i] += 1\n\ti += 1")] ... pybench2.runner(stmts, pythons, tracecmd)
(0, 0, "def f(x):\n\treturn x",
"res=[]\nfor x in 'spam' * 2500:\n\tres.append(f(x))")
C:\python33\python -m timeit -n 1000 -r 5 -s "def f(x):" -s " return x" "res=[]"
"for x in 'spam' * 2500:" " res.append(f(x))"c:\Python33\Lib\test>cd C:\python33\lib\testc:\Python33\Lib\test>py −3 pystone.pyPystone(1.1) time for 50000 passes = 0.685303 This machine benchmarks at 72960.4 pystones/second c:\Python33\Lib\test>cd c:\python27\lib\testc:\Python27\Lib\test>py −2 pystone.pyPystone(1.1) time for 50000 passes = 0.463547 This machine benchmarks at 107864 pystones/second c:\Python27\Lib\test>c:\pypy\pypy-1.9\pypy pystone.pyPystone(1.1) time for 50000 passes = 0.099975 This machine benchmarks at 500125 pystones/second
>>>X = 99>>>def selector():# X used but not assignedprint(X)# X found in global scope>>>selector()99
>>>def selector():print(X)# Does not yet exist!X = 88# X classified as a local name (everywhere)# Can also happen for "import X", "def X"...>>>selector()UnboundLocalError: local variable 'X' referenced before assignment
>>>def selector():global X# Force X to be global (everywhere)print(X)X = 88>>>selector()99
>>>X = 99>>>def selector():import __main__# Import enclosing moduleprint(__main__.X)# Qualify to get to global version of nameX = 88# Unqualified X classified as localprint(X)# Prints local version of name>>>selector()99 88
>>>def saver(x=[]):# Saves away a list objectx.append(1)# Changes same object each time!print(x)>>>saver([2])# Default not used[2, 1] >>>saver()# Default used[1] >>>saver()# Grows on each call![1, 1] >>>saver()[1, 1, 1]
>>>def saver(x=None):if x is None:# No argument passed?x = []# Run code to make a new list each timex.append(1)# Changes new list objectprint(x)>>>saver([2])[2, 1] >>>saver()# Doesn't grow here[1] >>>saver()[1]
>>>def saver():saver.x.append(1)print(saver.x)>>>saver.x = []>>>saver()[1] >>>saver()[1, 1] >>>saver()[1, 1, 1]
>>>def proc(x):print(x)# No return is a None return>>>x = proc('testing 123...')testing 123... >>>print(x)None
>>>list = [1, 2, 3]>>>list = list.append(4)# append is a "procedure">>>print(list)# append changes list in placeNone
map beats list
comprehensions in Python only when all tools must call functions;
for loops tend to be slower than
comprehensions; and generator functions and expressions are slower
than comprehensions by a constant factor. Under PyPy, some of these
findings differ; map often turns in
a different relative performance, for example, and list comprehensions
seem always quickest, perhaps due to function-level
optimizations.timer or
standard library timeit to test
your use cases for more relevant results. Also keep in mind that
iteration is just one component of a program’s time: more code gives a
more complete picture.timeit and
memory management differences may influence some results. The
pystone benchmark confirms these relative
rankings, though the sizes of the differences it reports differ due to
the code timed.timer or
standard library timeit to test
your use cases for more relevant results. This is especially true when
timing Python implementations, which may be arbitrarily optimized in
each new release.adder in a Python module file. The function
should accept two arguments and return the sum (or concatenation) of
the two. Then, add code at the bottom of the file to call the adder function with a variety of object
types (two strings, two lists, two floating points), and run this file
as a script from the system command line. Do you have to print the
call statement results to see results on your screen?adder function you wrote in the last
exercise to compute the sum of an arbitrary number of arguments, and
change the calls to pass more or fewer than two arguments. What type
is the return value sum? (Hints: a slice such as S[:0] returns an empty sequence of the same
type as S, and the type built-in function can test types; but
see the manually coded min examples
in Chapter 18 for a simpler approach.) What happens
if you pass in arguments of different types? What about passing in
dictionaries?adder function from exercise 2 to accept and
sum/concatenate three arguments: def
adder(good, bad, ugly). Now, provide default values for each
argument, and experiment with calling the function interactively. Try
passing one, two, three, and four arguments. Then, try passing keyword
arguments. Does the call adder(ugly=1,
good=2) work? Why? Finally, generalize the new adder to accept and sum/concatenate an
arbitrary number of keyword arguments. This is
similar to what you did in exercise 3, but you’ll need to iterate over
a dictionary, not a tuple. (Hint: the dict.keys method returns a list you can step
through with a for or while, but be sure to wrap it in a list call to index it in 3.X; dict.values may help here too.)copyDict(dict) that copies its
dictionary argument. It should return a new dictionary containing all
the items in its argument. Use the dictionary keys method to iterate (or, in Python 2.2
and later, step over a dictionary’s keys without calling keys). Copying sequences is easy (X[:] makes a top-level copy); does this work
for dictionaries, too? As explained in this exercise’s solution,
because dictionaries now come with similar tools, this and the next
exercise are just coding exercises but still serve as representative
function examples.addDict(dict1, dict2) that computes
the union of two dictionaries. It should return a new dictionary
containing all the items in both its arguments (which are assumed to
be dictionaries). If the same key appears in both arguments, feel free
to pick a value from either. Test your function by writing it in a
file and running the file as a script. What happens if you pass lists
instead of dictionaries? How could you generalize your function to
handle this case, too? (Hint: see the type built-in function used earlier.) Does
the order of the arguments passed in matter?def f1(a, b): print(a, b)# Normal argsdef f2(a, *b): print(a, b)# Positional varargsdef f3(a, **b): print(a, b)# Keyword varargsdef f4(a, *b, **c): print(a, b, c)# Mixed modesdef f5(a, b=2, c=3): print(a, b, c)# Defaultsdef f6(a, b=2, *c): print(a, b, c)# Defaults and positional varargs
>>>f1(1, 2)>>>f1(b=2, a=1)>>>f2(1, 2, 3)>>>f3(1, x=2, y=3)>>>f4(1, 2, 3, x=2, y=3) >>>f5(1)>>>f5(1, 4)>>>f6(1)>>>f6(1, 3, 4)
x = y // 2# For some y > 1while x > 1: if y % x == 0:# Remainderprint(y, 'has factor', x) break# Skip elsex -= 1 else:# Normal exitprint(y, 'is prime')
y should be a passed-in argument),
and add some calls to the function at the bottom of your file. While
you’re at it, experiment with replacing the first line’s // operator with / to see how true division changes the
/ operator in Python 3.X and breaks
this code (refer back to Chapter 5 if you need
a reminder). What can you do about negatives, and the values 0 and 1?
How about speeding this up? Your outputs should look something like
this:13 is prime 13.0 is prime 15 has factor 5 15.0 has factor 5.0
[2, 4, 9, 16, 25]. Code
this as a for loop first, then as a
map call, then as a list
comprehension, and finally as a generator expression. Use the sqrt function in the built-in math module to do the calculation (i.e.,
import math and say math.sqrt(x)). Of the four, which approach
do you like best?math.sqrt(X), X ** .5, and pow(X,
.5). If your programs run a lot of these, their relative
performance might become important. To see which is quickest,
repurpose the timerseqs.py script
we wrote in this chapter to time each of these three tools. Use the
bestof or bestoftotal functions in one of this
chapter’s timer modules to test
(you can use either the original, the 3.X-only keyword-only variant,
or the 2.X/3.X version, and may use Python’s timeit module as well). You might also want
to repackage the testing code in this script for better reusability — by
passing a test functions tuple to a general tester function, for
example (for this exercise a copy-and-modify approach is fine). Which
of the three square root tools seems to run fastest on your machine
and Python in general? Finally, how might you go about interactively
timing the speed of dictionary comprehensions versus for loops?countdown
that prints numbers as it counts down to zero. For example, a call
countdown(5) will print: 5 4 3 2 1 stop. There’s no obvious reason to
code this with an explicit stack or queue, but what about a
nonfunction approach? Would a generator make sense here?N!, computed as
N*(N-1)*(N-2)*...1. For instance,
6! is 6*5*4*3*2*1, or 720. Code and time four functions that, for
a call fact(N), each return
N!. Code these four functions (1)
as a recursive countdown per Chapter 19; (2) using the functional
reduce call per Chapter 19; (3) with a simple iterative
counter loop per Chapter 13; and (4) using
the math.factorial library tool per
Chapter 20. Use Chapter 21’s timeit to time each of your functions. What
conclusions can you draw from your results?1 A preview: notice how we must pass functions into the timer manually here. In Chapter 39 and Chapter 40 we’ll see decorator-based timer alternatives with which timed functions are called normally, but require extra “@” syntax where defined. Decorators may be more useful to instrument functions with timing logic when they are already being used within a larger system, and don’t as easily support the more isolated test call patterns assumed here — when decorated, every call to the function runs the timing logic, which is either a plus or minus depending on your goals.
As discussed in Chapter 3, modules let you save code in files permanently. Unlike code you type at the Python interactive prompt, which goes away when you exit Python, code in module files is persistent — it can be reloaded and rerun as many times as needed. Just as importantly, modules are a place to define names, known as attributes, which may be referenced by multiple external clients. When used well, this supports a modular program design that groups functionality into reusable units.
Modules are also the highest-level program organization unit in Python. Although they are fundamentally just packages of names, these packages are also self-contained — you can never see a name in another file, unless you explicitly import that file. Much like the local scopes of functions, this helps avoid name clashes across your programs. In fact, you can’t avoid this feature — everything “lives” in a module, both the code you run and the objects you create are always implicitly enclosed in modules. Because of that, modules are natural tools for grouping system components.
From an operational perspective, modules are also useful for implementing components that are shared across a system and hence require only a single copy. For instance, if you need to provide a global object that’s used by more than one function or file, you can code it in a module that can then be imported by many clients.
def spam(text): # File b.py
print(text, 'spam')import b# File a.pyb.spam('gumby')# Prints "gumby spam"
Load the file b.py (unless it’s already loaded), and give me access to all its attributes through the nameb.
Fetch the value of the namespamthat lives within the objectb.
If the byte code file is older than the source file (i.e., if you’ve changed the source) or was created by a different Python version, Python automatically regenerates the byte code when the program is run.As discussed ahead, this model is modified somewhat in Python 3.2 and later — byte code files are segregated in a __pycache__ subdirectory and named with their Python version to avoid contention and recompiles when multiple Pythons are installed. This obviates the need to check version numbers in the byte code, but the timestamp check is still used to detect changes in the source.
If, on the other hand, Python finds a .pyc byte code file that is not older than the corresponding .py source file and was created by the same Python version, it skips the source-to-byte-code compile step.In addition, if Python finds only a byte code file on the search path and no source, it simply loads the byte code directly; this means you can ship a program as just byte code files and avoid sending source. In other words, the compile step is bypassed if possible to speed program startup.
Byte code is stored in files in the same directory as the corresponding source files, normally with the filename extension .pyc (e.g., module.pyc). Byte code files are also stamped internally with the version of Python that created them (known as a “magic” field to developers) so Python knows to recompile when this differs in the version of Python running your program. For instance, if you upgrade to a new Python whose byte code differs, all your byte code files will be recompiled automatically due to a version number mismatch, even if you haven’t changed your source code.
Byte code is instead stored in files in a subdirectory named __pycache__, which Python creates if needed, and which is located in the directory containing the corresponding source files. This helps avoid clutter in your source directories by segregating the byte code files in their own directory. In addition, although byte code files still get the .pyc extension as before, they are given more descriptive names that include text identifying the version of Python that created them (e.g., module.cpython-32.pyc). This avoids contention and recompiles: because each version of Python installed can have its own uniquely named version of byte code files in the __pycache__ subdirectory, running under a given version doesn’t overwrite the byte code of another, and doesn’t require recompiles. Technically, byte code filenames also include the name of the Python that created them, so CPython, Jython, and other implementations mentioned in the preface and Chapter 2 can coexist on the same machine without stepping on each other’s work (once they support this model).
c:\code\py2x>dir10/31/2012 10:58 AM 39 script0.py c:\code\py2x>C:\python27\python>>>import script0hello world 1267650600228229401496703205376 >>>^Zc:\code\py2x>dir10/31/2012 10:58 AM 39 script0.py 10/31/2012 11:00 AM 154 script0.pyc
c:\code\py2x>cd ..\py3xc:\code\py3x>dir10/31/2012 10:58 AM 39 script0.py c:\code\py3x>C:\python33\python>>>import script0hello world 1267650600228229401496703205376 >>>^Zc:\code\py3x>dir10/31/2012 10:58 AM 39 script0.py 10/31/2012 11:00 AM <DIR> __pycache__ c:\code\py3x>dir __pycache__10/31/2012 11:00 AM 184 script0.cpython-33.pyc
c:\code\py3x>C:\python32\python>>>import script0hello world 1267650600228229401496703205376 >>>^Zc:\code\py3x>dir __pycache__10/31/2012 12:28 PM 178 script0.cpython-32.pyc 10/31/2012 11:00 AM 184 script0.cpython-33.pyc
PYTHONPATH directories (if
set)Python first looks for the imported file in the home directory. The meaning of this entry depends on how you are running the code. When you’re running a program, this entry is the directory containing your program’s top-level script file. When you’re working interactively, this entry is the directory in which you are working (i.e., the current working directory).Because this directory is always searched first, if a program is located entirely in a single directory, all of its imports will work automatically with no path configuration required. On the other hand, because this directory is searched first, its files will also override modules of the same name in directories elsewhere on the path; be careful not to accidentally hide library modules this way if you need them in your program, or use package tools we’ll meet later that can partially sidestep this issue.
PYTHONPATH directories
(configurable)Next, Python searches all directories listed in yourPYTHONPATHenvironment variable setting, from left to right (assuming you have set this at all: it’s not preset for you). In brief,PYTHONPATHis simply a list of user-defined and platform-specific names of directories that contain Python code files. You can add all the directories from which you wish to be able to import, and Python will extend the module search path to include all the directories yourPYTHONPATHlists.Because Python searches the home directory first, this setting is only important when importing files across directory boundaries — that is, if you need to import a file that is stored in a different directory from the file that imports it. You’ll probably want to set yourPYTHONPATHvariable once you start writing substantial programs, but when you’re first starting out, as long as you save all your module files in the directory in which you’re working (i.e., the home directory, like the C:\code used in this book) your imports will work without you needing to worry about this setting at all.
Next, Python automatically searches the directories where the standard library modules are installed on your machine. Because these are always searched, they normally do not need to be added to yourPYTHONPATHor included in path files (discussed next).
Next, a lesser-used feature of Python allows users to add directories to the module search path by simply listing them, one per line, in a text file whose name ends with a .pth suffix (for “path”). These path configuration files are a somewhat advanced installation-related feature; we won’t cover them fully here, but they provide an alternative toPYTHONPATHsettings.In short, text files of directory names dropped in an appropriate directory can serve roughly the same role as thePYTHONPATHenvironment variable setting. For instance, if you’re running Windows and Python 3.3, a file named myconfig.pth may be placed at the top level of the Python install directory (C:\Python33) or in the site-packages subdirectory of the standard library there (C:\Python33\Lib\site-packages) to extend the module search path. On Unix-like systems, this file might be located in /usr/local/lib/python3.3/site-packages or /usr/local/lib/site-python instead.When such a file is present, Python will add the directories listed on each line of the file, from first to last, near the end of the module search path list — currently, afterPYTHONPATHand standard libraries, but before the site-packages directory where third-party extensions are often installed. In fact, Python will collect the directory names in all the .pth path files it finds and will filter out any duplicates and nonexistent directories. Because they are files rather than shell settings, path files can apply to all users of an installation, instead of just one user or shell. Moreover, for some users and applications, text files may be simpler to code than environment settings.This feature is more sophisticated than I’ve described here. For more details, consult the Python library manual, and especially its documentation for the standard library modulesite— this module allows the locations of Python libraries and path files to be configured, and its documentation describes the expected locations of path files in general. I recommend that beginners usePYTHONPATHor perhaps a single .pth file, and then only if you must import across directories. Path files are used more often by third-party libraries, which commonly install a path file in Python’s site-packages, described next.
Finally, Python automatically adds the site-packages subdirectory of its standard library to the module search path. By convention, this is the place that most third-party extensions are installed, often automatically by thedistutilsutility described in an upcoming sidebar. Because their install directory is always part of the module search path, clients can import the modules of such extensions without any path settings.
c:\pycode\utilities;d:\pycode\package1
c:\pycode\utilities d:\pycode\package1
>>>import sys>>>sys.path['', 'C:\\code', 'C:\\Windows\\system32\\python33.zip', 'C:\\Python33\\DLLs', 'C:\\Python33\\lib', 'C:\\Python33', 'C:\\Users\\mark', 'C:\\Python33\\lib\\site-packages']
PYTHONPATH environment variable?PYTHONPATH to import from directories other
than the one in which you are working (i.e., the current directory
when working interactively, or the directory containing your top-level
file). In practice, this will be a common case for nontrivial
programs.PYTHONPATH environment variable, the
standard library directories, all directories listed in .pth path files located in standard places,
and the site-packages root
directory for third-party extension installs. Of these, programmers
can customize PYTHONPATH and
.pth files.def or class statements). Technically, a module’s
global scope morphs into the module object’s
attributes namespace. A module’s namespace may also be altered by
assignments from other files that import it, though this is generally
frowned upon (see Chapter 17 for more on the downsides
of cross-file changes).1 It’s syntactically illegal to include path and extension
details in a standard import.
However, package imports, which we’ll discuss
in Chapter 24, allow import statements to include part of the
directory path leading to a file as a set of period-separated names.
Package imports, though, still rely on the normal module search path
to locate the leftmost directory in a package path (i.e., they are
relative to a directory in the search path). They also cannot make
use of any platform-specific directory syntax in the import statements; such syntax only works
on the search path. Also, note that module file search path issues
are not as relevant when you run frozen
executables (discussed in Chapter 2), which typically embed byte
code in the binary image.
2 As described earlier, Python keeps already imported modules in
the built-in sys.modules
dictionary so it can keep track of what’s been loaded. In fact, if
you want to see which modules are loaded, you can import sys and print list(sys.modules.keys()). There’s more on
other uses for this internal table in Chapter 25.
3 Also watch for Chapter 24’s discussion
of the new relative import syntax and search
rules in Python 3.X; they modify the search path for from statements in files inside packages
when “.” characters are used (e.g., from .
import string). By default, a package’s own directory is
not automatically searched by imports in Python 3.X, unless such
relative imports are used by files in the package itself.
def printer(x): # Module attribute
print(x)>>>import module1# Get module as a whole (one or more)>>>module1.printer('Hello world!')# Qualify to get namesHello world!
>>>from module1 import printer# Copy out a variable (one or more)>>>printer('Hello world!')# No need to qualify nameHello world!
>>>from module1 import *# Copy out _all_ variables>>>printer('Hello world!')Hello world!
print('hello')
spam = 1 # Initialize variable%python>>>import simple# First import: loads and runs file's codehello >>>simple.spam# Assignment makes an attribute1
>>>simple.spam = 2# Change attribute in module>>>import simple# Just fetches already loaded module>>>simple.spam# Code wasn't rerun: attribute unchanged2
import assigns an entire
module object to a single name.from assigns one or more
names to objects of the same names in another module.x = 1 y = [1, 2]
%python>>>from small import x, y# Copy two names out>>>x = 42# Changes local x only>>>y[0] = 42# Changes shared mutable in place
>>>import small# Get module name (from doesn't)>>>small.x# Small's x is not my x1 >>>small.y# But we share a changed mutable[42, 2]
%python>>>from small import x, y# Copy two names out>>>x = 42# Changes my x only>>>import small# Get module name>>>small.x = 42# Changes x in other module
from module import name1, name2 # Copy these two names out (only)import module# Fetch the module objectname1 = module.name1# Copy names out by assignmentname2 = module.name2 del module# Get rid of the module name
# M.pydef func():...do something...# N.pydef func():...do something else...
# O.pyfrom M import func from N import func# This overwrites the one we fetched from Mfunc()# Calls N.func only!
# O.pyimport M, N# Get the whole modules, not their namesM.func()# We can call both names nowN.func()# The module names make them unique
# O.pyfrom M import func as mfunc# Rename uniquely with "as"from N import func as nfunc mfunc(); nfunc()# Calls one or the other
def
or class that assign names (e.g.,
=, def) create attributes of the module
object; assigned names are stored in the module’s namespace.__dict__ or dir(M). Module namespaces created by
imports are dictionaries; they may be accessed through the built-in
__dict__ attribute associated
with module objects and may be inspected with the dir function.
The dir function is roughly
equivalent to the sorted keys list of an object’s __dict__
attribute, but it includes inherited names for classes, may not be
complete, and is prone to changing from release to release.print('starting to load...')
import sys
name = 42
def func(): pass
class klass: pass
print('done loading.')>>> import module2
starting to load...
done loading.>>>module2.sys<module 'sys' (built-in)> >>>module2.name42 >>>module2.func<function func at 0x000000000222E7B8> >>>module2.klass<class 'module2.klass'>
>>> list(module2.__dict__.keys())
['__loader__', 'func', 'klass', '__builtins__', '__doc__', '__file__', '__name__',
'name', '__package__', 'sys', '__initializing__', '__cached__']>>>list(name for name in module2.__dict__.keys() if not name.startswith('__'))['func', 'klass', 'name', 'sys'] >>>list(name for name in module2.__dict__ if not name.startswith('__'))['func', 'sys', 'name', 'klass']
>>> module2.name, module2.__dict__['name']
(42, 42)X.Ymeans findXin the current scopes, then search for the attributeYin the objectX(not in scopes).
X.Y.Zmeans look up the nameYin the objectX, then look upZin the objectX.Y.
Qualification works on all objects with attributes: modules, classes, C extension types, etc.
X = 88# My X: global to this file onlydef f(): global X# Change this file's XX = 99# Cannot see names in other modules
X = 11# My X: global to this file onlyimport moda# Gain access to names in modamoda.f()# Sets moda.X, not this file's Xprint(X, moda.X)
% python modb.py
11 99X = 3
X = 2 import mod3 print(X, end=' ')# My global Xprint(mod3.X)# mod3's X
X = 1 import mod2 print(X, end=' ')# My global Xprint(mod2.X, end=' ')# mod2's Xprint(mod2.mod3.X)# Nested mod3's X
% python mod1.py
2 3
1 2 3import and
from statements) load and run a
module’s code only the first time the module is imported in a
process.reload function forces an
already loaded module’s code to be reloaded and rerun. Assignments in
the file’s new code change the existing module object in place.reload is a function in
Python, not a statement.reload is passed an
existing module object, not a new name.reload lives in a module in
Python 3.X and must be imported itself.import module# Initial import...use module.attributes... ...# Now, go change the module file... from imp import reload# Get reload itself (in 3.X)reload(module)# Get updated exports...use module.attributes...
reload runs a module file’s new code in the module’s current
namespace. Rerunning a module file’s code overwrites its
existing namespace, rather than deleting and re-creating it.def statement replaces the prior
version of the function in the module’s namespace by reassigning the
function name.import
to fetch modules. Because clients
that use import qualify to fetch
attributes, they’ll find new values in the module object after a
reload.from clients only. Clients that used from to fetch attributes in the past won’t
be affected by a reload; they’ll still have references to the old
objects fetched before the reload.message = "First version"
def printer():
print(message)%python>>>import changer>>>changer.printer()First version
...modify changer.py without stopping Python...%notepad changer.py
message = "After editing"
def printer():
print('reloaded:', message)...back to the Python interpreter...>>>import changer>>>changer.printer()# No effect: uses loaded moduleFirst version >>>from imp import reload>>>reload(changer)# Forces new code to load/run<module 'changer' from '.\\changer.py'> >>>changer.printer()# Runs the new version nowreloaded: After editing
from statement imports an
entire module, like the import
statement, but as an extra step it also copies one or more variables
from the imported module into the scope where the from appears. This enables you to use the
imported names directly (name)
instead of having to go through the module (module.name).reload function forces a module to
be imported again. It is mostly used to pick up new versions of a
module’s source code during development, and in dynamic customization
scenarios.import instead
of from only when you need to
access the same name in two different modules; because you’ll have to
specify the names of the enclosing modules, the two names will be
unique. The as extension can render
from usable in this context as
well.from statement can
obscure the meaning of a variable (which module it is defined in), can
have problems with the reload call
(names may reference prior versions of objects), and can corrupt
namespaces (it might silently overwrite names you are using in your
scope). The from * form is worse in
most regards — it can seriously corrupt namespaces and obscure the
meaning of variables, so it is probably best used sparingly.1 Some languages act differently and provide for dynamic scoping, where scopes really may depend on runtime calls. This tends to make code trickier, though, because the meaning of a variable can differ over time. In Python, scopes more simply correspond to the text of your program.
import dir1.dir2.mod
from dir1.dir2.mod import x
dir0\dir1\dir2\mod.py # Or mod.pyc, mod.so, etc.import C:\mycode\dir1\dir2\mod # Error: illegal syntaximport dir1.dir2.mod
dir0\dir1\dir2\mod.py
import dir1.dir2.mod
sys.path.dir0\ # Container on module search path
dir1\
__init__.py
dir2\
__init__.py
mod.pyThe first time a Python program imports through a directory, it automatically runs all the code in the directory’s __init__.py file. Because of that, these files are a natural place to put code to initialize the state required by files in a package. For instance, a package might use its initialization file to create required data files, open connections to databases, and so on. Typically, __init__.py files are not meant to be useful if executed directly; they are run automatically when a package is first accessed.
Package __init__.py files are also partly present to declare that a directory is a Python package. In this role, these files serve to prevent directories with common names from unintentionally hiding true modules that appear later on the module search path. Without this safeguard, Python might pick a directory that has nothing to do with your code, just because it appears nested in an earlier directory on the search path. As we’ll see later, Python 3.3’s namespace packages obviate much of this role, but achieve a similar effect algorithmically by scanning ahead on the path to find later files.
In the package import model, the directory paths in your script become real nested object paths after an import. For instance, in the preceding example, after the import the expressiondir1.dir2works and returns a module object whose namespace contains all the names assigned by dir2’s __init__.py initialization file. Such files provide a namespace for module objects created for directories, which would otherwise have no real associated module file.
from * statement
behaviorAs an advanced feature, you can use__all__lists in __init__.py files to define what is exported when a directory is imported with thefrom *statement form. In an __init__.py file, the__all__list is taken to be the list of submodule names that should be automatically imported whenfrom *is used on the package (directory) name. If__all__is not set, thefrom *statement does not automatically load submodules nested in the directory; instead, it loads just names defined by assignments in the directory’s __init__.py file, including any submodules explicitly imported by code in this file. For instance, the statementfrom submodule import Xin a directory’s __init__.py makes the nameXavailable in that directory’s namespace. (We’ll see additional roles for__all__in Chapter 25: it serves to declarefrom *exports of simple files as well.)
# dir1\__init__.pyprint('dir1 init') x = 1# dir1\dir2\__init__.pyprint('dir2 init') y = 2# dir1\dir2\mod.pyprint('in mod.py') z = 3
C:\code>python# Run in dir1's container directory>>>import dir1.dir2.mod# First imports run init filesdir1 init dir2 init in mod.py >>> >>>import dir1.dir2.mod# Later imports do not
>>>from imp import reload# from needed in 3.X only>>>reload(dir1)dir1 init <module 'dir1' from '.\\dir1\\__init__.py'> >>> >>>reload(dir1.dir2)dir2 init <module 'dir1.dir2' from '.\\dir1\\dir2\\__init__.py'>
>>>dir1<module 'dir1' from '.\\dir1\\__init__.py'> >>>dir1.dir2<module 'dir1.dir2' from '.\\dir1\\dir2\\__init__.py'> >>>dir1.dir2.mod<module 'dir1.dir2.mod' from '.\\dir1\\dir2\\mod.py'>
>>>dir1.x1 >>>dir1.dir2.y2 >>>dir1.dir2.mod.z3
>>>dir2.modNameError: name 'dir2' is not defined >>>mod.zNameError: name 'mod' is not defined
C:\code>python>>>from dir1.dir2 import mod# Code path here onlydir1 init dir2 init in mod.py >>>mod.z# Don't repeat path3 >>>from dir1.dir2.mod import z>>>z3 >>>import dir1.dir2.mod as mod# Use shorter name (see Chapter 25)>>>mod.z3 >>>from dir1.dir2.mod import z as modz# Ditto if names clash (see Chapter 25)>>>modz3
import utilities
import database.client.utilities
system1\
utilities.py # Common utility functions, classes
main.py # Launch this to start the program
other.py # Import utilities to load my toolssystem2\
utilities.py # Common utilities
main.py # Launch this to run
other.py # Imports utilitiesimport utilities
utilities.func('spam')root\
system1\
__init__.py
utilities.py
main.py
other.py
system2\
__init__.py
utilities.py
main.py
other.py
system3\ # Here or elsewhere
__init__.py # Need __init__.py here only if imported elsewhere
myfile.py # Your new code hereimport system1.utilities
import system2.utilities
system1.utilities.function('spam')
system2.utilities.function('eggs')sys.path search path. These
are known as absolute imports.from statements to allow them to
explicitly request that imports search the package’s directory only,
with leading dots. This is known as relative
import syntax.from statements’ module names to indicate
that imports should be relative-only to the
containing package — such imports will search for modules inside the
package directory only and will not look for same-named modules
located elsewhere on the import search path (sys.path). The net effect is that package
modules override outside modules.sys.path
search path.from . import spam # Relative to this packagefrom .spam import name
from __future__ import absolute_import # Use 3.X relative import model in 2.Ximport string # Skip this package's versionfrom . import string # Searches this package onlyfrom .string import name1, name2# Imports names from mypkg.stringfrom . import string# Imports mypkg.stringfrom .. import string# Imports string sibling of mypkg
mypkg\
__init__.py
main.py
string.pyimport string # Imports string outside package (absolute)from string import name # Imports name from string outside packagefrom . import string # Imports mypkg.string here (relative)from .string import name1, name2 # Imports names from mypkg.stringfrom .. import spam # Imports a sibling of mypkgfrom . import D# Imports A.B.D (. means A.B)from .. import E# Imports A.E (.. means A)from .D import X# Imports A.B.D.X (. means A.B)from ..E import X# Imports A.E.X (.. means A)
from mypkg import string # Imports mypkg.string (absolute)from system.section.mypkg import string # system container on sys.path onlyfrom . import string # Relative import syntaxfrom statement only. Also remember that this
feature’s new syntax applies only to from statements, not import statements. It’s detected by the
fact that the module name in a from begins with one or more dots
(periods). Module names that contain embedded dots but don’t have a
leading dot are package imports, not relative imports.A) are located by searching each directory
on the sys.path list, from left
to right. This list is constructed from both system defaults and
user-configurable settings described in Chapter 22.A.B.C directory path
syntax in imports. In an import of A.B.C, for example, the directory named
A is located relative to the
normal module import search of sys.path, B is another package subdirectory within
A, and C is a module or other importable item
within B.import and from statements use the same sys.path
search rule as imports elsewhere. Imports in packages using from statements and leading
dots, however, are relative to the package;
that is, only the package directory is checked, and the normal
sys.path lookup is not used. In
from . import A, for example, the
module search is restricted to the directory containing the file in
which this statement appears.from . import
m, from .m import xAre relative-only in both 2.X and 3.X
import
m, from m import
xAre relative-then-absolute in 2.X, and absolute-only in 3.X
C:\code>c:\Python33\python>>>import string>>>string<module 'string' from 'C:\\Python33\\lib\\string.py'>
# code\string.pyprint('string' * 8) C:\code>c:\Python33\python>>>import stringstringstringstringstringstringstringstringstring >>>string<module 'string' from '.\\string.py'>
>>> from . import string
SystemError: Parent module '' not loaded, cannot perform relative import# code\main.pyimport string# Same code but in a fileprint(string) C:\code>C:\python33\python main.py# Equivalent results in 2.Xstringstringstringstringstringstringstringstring <module 'string' from 'c:\\code\\string.py'>
C:\code>del string*# del __pycache__\string* for bytecode in 3.2+C:\code>mkdir pkgc:\code>notepad pkg\__init__.py# code\pkg\spam.pyimport eggs# <== Works in 2.X but not 3.X!print(eggs.X)# code\pkg\eggs.pyX = 99999 import string print(string)
C:\code>c:\Python27\python>>>import pkg.spam<module 'string' from 'C:\Python27\lib\string.pyc'> 99999 C:\code>c:\Python33\python>>>import pkg.spamImportError: No module named 'eggs'
# code\pkg\spam.pyfrom . import eggs# <== Use package relative import in 2.X or 3.Xprint(eggs.X)# code\pkg\eggs.pyX = 99999 import string print(string) C:\code>c:\Python27\python>>>import pkg.spam<module 'string' from 'C:\Python27\lib\string.pyc'> 99999 C:\code>c:\Python33\python>>>import pkg.spam<module 'string' from 'C:\\Python33\\lib\\string.py'> 99999
# code\string.pyprint('string' * 8)# code\pkg\spam.pyfrom . import eggs print(eggs.X)# code\pkg\eggs.pyX = 99999 import string# <== Gets string in CWD, not Python lib!print(string) C:\code>c:\Python33\python# Same result in 2.X>>>import pkg.spamstringstringstringstringstringstringstringstring <module 'string' from '.\\string.py'> 99999
C:\code>del string*# del __pycache__\string* for bytecode in 3.2+# code\pkg\spam.pyimport string# <== Relative in 2.X, absolute in 3.Xprint(string)# code\pkg\string.pyprint('Ni' * 8)
C:\code>c:\Python33\python>>>import pkg.spam<module 'string' from 'C:\\Python33\\lib\\string.py'> C:\code>c:\Python27\python>>>import pkg.spamNiNiNiNiNiNiNiNi <module 'pkg.string' from 'pkg\string.py'>
# code\pkg\spam.pyfrom . import string# <== Relative in both 2.X and 3.Xprint(string)# code\pkg\string.pyprint('Ni' * 8) C:\code>c:\Python33\python>>>import pkg.spamNiNiNiNiNiNiNiNi <module 'pkg.string' from '.\\pkg\\string.py'> C:\code>c:\Python27\python>>>import pkg.spamNiNiNiNiNiNiNiNi <module 'pkg.string' from 'pkg\string.py'>
# code\pkg\spam.pyfrom . import string# <== Fails in both 2.X and 3.X if no string.py here!C:\code>del pkg\string*C:\code>C:\python33\python>>>import pkg.spamImportError: cannot import name string C:\code>C:\python27\python>>>import pkg.spamImportError: cannot import name string
# code\string.pyprint('string' * 8)# code\pkg\spam.pyfrom . import string# <== Relative in both 2.X and 3.Xprint(string)# code\pkg\string.pyprint('Ni' * 8)
C:\code>c:\Python33\python# Same result in 2.X>>>import pkg.spamNiNiNiNiNiNiNiNi <module 'pkg.string' from '.\\pkg\\string.py'>
# code\string.pyprint('string' * 8)# code\pkg\spam.pyimport string# <== Relative in 2.X, "absolute" in 3.X: CWD!print(string)# code\pkg\string.pyprint('Ni' * 8) C:\code>c:\Python33\python>>>import pkg.spamstringstringstringstringstringstringstringstring <module 'string' from '.\\string.py'> C:\code>c:\Python27\python>>>import pkg.spamNiNiNiNiNiNiNiNi <module 'pkg.string' from 'pkg\string.pyc'>
from
. relative syntax to be used unless the importer is being
used as part of a package (i.e., is being imported from somewhere
else).from .
relative syntax is used (or the module is in the current working
directory or main script’s home directory).from . import mod# Not allowed in nonpackage mode in both 2.X and 3.Ximport mod# Does not search file's own directory in package mode in 3.X
from system.section.mypkg import mod # Works in both program and package mode# code\pkg\main.pyimport spam# code\pkg\spam.pyimport eggs# <== Works if in "." = home of main script file# code\pkg\eggs.pyprint('Eggs' * 4)# But won't load this file when used as pkg in 3.X!c:\code>python pkg\main.py# OK as program, in both 2.X and 3.XEggsEggsEggsEggs c:\code>python pkg\spam.pyEggsEggsEggsEggs c:\code>py −2# OK as package in 2.X: relative-then-absolute>>>import pkg.spam# 2.X: plain imports search package directory firstEggsEggsEggsEggs C:\code>py −3# But 3.X fails to find file here: absolute only>>>import pkg.spam# 3.X: plain imports search only CWD plus sys.pathImportError: No module named 'eggs'
# code\pkg\main.pyimport spam# code\pkg\spam.pyfrom . import eggs# <== Not a package if main file here (even if me)!# code\pkg\eggs.pyprint('Eggs' * 4) c:\code>python# OK as package but not program in both 3.X and 2.X>>>import pkg.spamEggsEggsEggsEggs c:\code>python pkg\main.pySystemError: ... cannot perform relative import c:\code>python pkg\spam.pySystemError: ... cannot perform relative import
# code\pkg\main.pyimport sub.spam# <== Works if move modules to pkg below main file# code\pkg\sub\spam.pyfrom . import eggs# Package relative works now: in subdirectory# code\pkg\sub\eggs.pyprint('Eggs' * 4) c:\code>python pkg\main.py# From main script: same result in 2.X and 3.XEggsEggsEggsEggs c:\code>python# From elsewhere: same result in 2.X and 3.X>>>import pkg.sub.spamEggsEggsEggsEggs
c:\code>py −3 pkg\sub\spam.py# But individual modules can't be run to testSystemError: ... cannot perform relative import
# code\pkg\main.pyimport spam# code\pkg\spam.pyimport pkg.eggs# <== Full package paths work in all cases, 2.X+3.X# code\pkg\eggs.pyprint('Eggs' * 4) c:\code>set PYTHONPATH=C:\codec:\code>python pkg\main.py# From main script: Same result in 2.X and 3.XEggsEggsEggsEggs c:\code>python# From elsewhere: Same result in 2.X and 3.X>>>import pkg.spamEggsEggsEggsEggs
c:\code>python pkg\spam.py# Individual modules are runnable too in 2.X and 3.XEggsEggsEggsEggs
# code\dualpkg\m1.pydef somefunc(): print('m1.somefunc')# code\dualpkg\m2.py...import m1 here...# Replace me with a real import statementdef somefunc(): m1.somefunc() print('m2.somefunc') if __name__ == '__main__': somefunc()# Self-test or top-level script usage mode code
# code\dualpkg\m2.pyfrom . import m1 c:\code>py −3>>>import dualpkg.m2# OKC:\code>py −2>>>import dualpkg.m2# OKc:\code>py −3 dualpkg\m2.py# Fails!c:\code>py −2 dualpkg\m2.py# Fails!
# code\dualpkg\m2.pyimport m1 c:\code>py −3>>>import dualpkg.m2# Fails!c:\code>py −2>>>import dualpkg.m2# OKc:\code>py −3 dualpkg\m2.py# OKc:\code>py −2 dualpkg\m2.py# OK
# code\dualpkg\m2.pyimport dualpkg.m1 as m1 # And: set PYTHONPATH=c:\code c:\code>py −3>>>import dualpkg.m2# OKC:\code>py −2>>>import dualpkg.m2# OKc:\code>py −3 dualpkg\m2.py# OKc:\code>py −2 dualpkg\m2.py# OK
import
mod, from mod import
attrThe original model: imports of files and their contents, relative to thesys.pathmodule search path
import
dir1.dir2.mod, from dir1.mod import
attrImports that give directory path extensions relative to thesys.pathmodule search path, where each package is contained in a single directory and has an initialization file, in Python 2.X and 3.X
from . import
mod (relative), import
mod (absolute)The model used for intrapackage imports of the prior section, with its relative or absolute lookup schemes for dotted and nondotted imports, available but differing in Python 2.X and 3.X
import
splitdir.modThe new namespace package model that we’ll survey here, which allows packages to span multiple directories, and requires no initialization file, introduced in Python 3.3
directory\spam\__init__.py is found, a regular
package is imported and returned.directory\spam.{py, pyc, or other module
extension} is found, a simple module is imported and
returned.directory\spam is found and is a directory, it is
recorded and the scan continues with the next directory in the
search path.C:\code\ns\dir1\sub\mod1.py C:\code\ns\dir2\sub\mod2.py
c:\code>mkdir ns\dir1\sub# Two dirs of same name in different dirsc:\code>mkdir ns\dir2\sub# And similar outside Windowsc:\code>type ns\dir1\sub\mod1.py# Module files in different directoriesprint(r'dir1\sub\mod1') c:\code>type ns\dir2\sub\mod2.pyprint(r'dir2\sub\mod2') c:\code>set PYTHONPATH=C:\code\ns\dir1;C:\code\ns\dir2
c:\code>C:\Python33\python>>>import sub>>>sub# Namespace packages: nested search paths<module 'sub' (namespace)> >>>sub.__path___NamespacePath(['C:\\code\\ns\\dir1\\sub', 'C:\\code\\ns\\dir2\\sub']) >>>from sub import mod1dir1\sub\mod1 >>>import sub.mod2# Content from two different directoriesdir2\sub\mod2 >>>mod1<module 'sub.mod1' from 'C:\\code\\ns\\dir1\\sub\\mod1.py'> >>>sub.mod2<module 'sub.mod2' from 'C:\\code\\ns\\dir2\\sub\\mod2.py'>
c:\code>C:\Python33\python>>>import sub.mod1dir1\sub\mod1 >>>import sub.mod2# One package spanning two directoriesdir2\sub\mod2 >>>sub.mod1<module 'sub.mod1' from 'C:\\code\\ns\\dir1\\sub\\mod1.py'> >>>sub.mod2<module 'sub.mod2' from 'C:\\code\\ns\\dir2\\sub\\mod2.py'> >>>sub<module 'sub' (namespace)> >>>sub.__path___NamespacePath(['C:\\code\\ns\\dir1\\sub', 'C:\\code\\ns\\dir2\\sub'])
c:\code>type ns\dir1\sub\mod1.pyfrom . import mod2# And "from . import string" still failsprint(r'dir1\sub\mod1') c:\code>C:\Python33\python>>>import sub.mod1# Relative import of mod2 in another dirdir2\sub\mod2 dir1\sub\mod1 >>>import sub.mod2# Already imported module not rerun>>>sub.mod2<module 'sub.mod2' from 'C:\\code\\ns\\dir2\\sub\\mod2.py'>
c:\code>mkdir ns\dir2\sub\lower# Further nested componentsc:\code>type ns\dir2\sub\lower\mod3.pyprint(r'dir2\sub\lower\mod3') c:\code>C:\Python33\python>>>import sub.lower.mod3# Namespace pkg nested in namespace pkgdir2\sub\lower\mod3 c:\code>C:\Python33\python>>>import sub# Same effect if accessed incrementally>>>import sub.mod2dir2\sub\mod2 >>>import sub.lower.mod3dir2\sub\lower\mod3 >>>sub.lower# A single-directory namespace pkg<module 'sub.lower' (namespace)> >>>sub.lower.__path___NamespacePath(['C:\\code\\ns\\dir2\\sub\\lower'])
c:\code>mkdir ns\dir1\sub\pkgC:\code>type ns\dir1\sub\pkg\__init__.pyprint(r'dir1\sub\pkg\__init__.py') c:\code>C:\Python33\python>>>import sub.mod2# Nested moduledir2\sub\mod2 >>>import sub.pkg# Nested regular packagedir1\sub\pkg\__init__.py >>>import sub.lower.mod3# Nested namespace packagedir2\sub\lower\mod3 >>>sub# Modules, packages,and namespaces<module 'sub' (namespace)> >>>sub.mod2<module 'sub.mod2' from 'C:\\code\\ns\\dir2\\sub\\mod2.py'> >>>sub.pkg<module 'sub.pkg' from 'C:\\code\\ns\\dir1\\sub\\pkg\\__init__.py'> >>>sub.lower<module 'sub.lower' (namespace)> >>>sub.lower.mod3<module 'sub.lower.mod3' from 'C:\\code\\ns\\dir2\\sub\\lower\\mod3.py'>
c:\code>mkdir ns2c:\code>mkdir ns3c:\code>mkdir ns3\dirc:\code>notepad ns3\dir\ns2.pyc:\code>type ns3\dir\ns2.pyprint(r'ns3\dir\ns2.py!')
c:\code>set PYTHONPATH=c:\code>py −3.2>>>import ns2ImportError: No module named ns2 c:\code>py −3.3>>>import ns2>>>ns2# A single-directory namespace package in CWD<module 'ns2' (namespace)> >>>ns2.__path___NamespacePath(['.\\ns2'])
c:\code>set PYTHONPATH=C:\code\ns3\dirc:\code>py −3.3>>>import ns2# Use later module file, not same-named directory!ns3\dir\ns2.py! >>>ns2<module 'ns2' from 'C:\\code\\ns3\\dir\\ns2.py'> >>>import sys>>>sys.path[:2]# First '' means current working directory, CWD['', 'C:\\code\\ns3\\dir']
c:\code>py −3.2>>>import ns2ns3\dir\ns2.py! >>>ns2<module 'ns2' from 'C:\code\ns3\dir\ns2.py'>
c:\code>mkdir ns4\dir1\subc:\code>mkdir ns4\dir2\subc:\code>set PYTHONPATH=c:\code\ns4\dir1;c:\code\ns4\dir2c:\code>py −3>>>import sub>>>sub<module 'sub' (namespace)> >>>sub.__path___NamespacePath(['c:\\code\\ns4\\dir1\\sub', 'c:\\code\\ns4\\dir2\\sub'])
c:\code>notepad ns4\dir2\sub\__init__.pyc:\code>py −3>>>import sub# Use later reg. package, not same-named directory!>>>sub<module 'sub' from 'c:\\code\\ns4\\dir2\\sub\\__init__.py'>
import
instead of from with
packages?from
mypkg import spam and from . import
spam?from statement with a
package to copy names out of the package directly, or use the as extension with the import statement to rename the path to a
shorter synonym. In both cases, the path is listed in only one place,
in the from or import statement.import or from statement must contain an __init__.py file. Other directories,
including the directory that contains the leftmost component of a
package path, do not need to include this file.import instead
of from with packages only if you
need to access the same name defined in more than one path. With
import, the path makes the
references unique, but from allows
only one version of any given name (unless you also use the as extension to rename).from mypkg import
spam is an absolute import — the search
for mypkg skips the package
directory and the module is located in an absolute directory in
sys.path. A statement from . import spam, on the other hand, is a
relative import — spam is looked up relative to the package in
which this statement is contained only. In Python 2.X, the absolute
import searches the package directory first before proceeding to
sys.path; relative imports work as
described.1 The dot path syntax was chosen partly for platform neutrality,
but also because paths in import
statements become real nested object paths. This syntax also means
that you may get odd error messages if you forget to omit the
.py in your import statements. For example, import mod.py is assumed to be a directory
path import — it loads mod.py,
then tries to load a mod\py.py,
and ultimately issues a potentially confusing “No module named py”
error message. As of Python 3.3 this error message has been improved
to say “No module named ‘mod.py’; mod is not a package.”
__main__; the only unique things about the
interactive prompt are that code runs and is discarded immediately,
and expression results are printed automatically.# unders.pya, _b, c, _d = 1, 2, 3, 4 >>>from unders import *# Load non _X names only>>>a, c(1, 3) >>>_bNameError: name '_b' is not defined >>>import unders# But other importers get every name>>>unders._b2
# alls.py__all__ = ['a', '_c']# __all__ has precedence over _Xa, b, _c, _d = 1, 2, 3, 4 >>>from alls import *# Load __all__ names only>>>a, _c(1, 3) >>>bNameError: name 'b' is not defined >>>from alls import a, b, _c, _d# But other importers get every name>>>a, b, _c, _d(1, 2, 3, 4) >>>import alls>>>alls.a, alls.b, alls._c, alls._d(1, 2, 3, 4)
from __future__ import featurename__name__ is set to the string "__main__" when it starts.__name__ is set to the module’s name as
known by its clients.def tester():
print("It's Christmas in Heaven...")
if __name__ == '__main__': # Only when run
tester() # Not when importedc:\code>python>>>import runme>>>runme.tester()It's Christmas in Heaven...
c:\code> python runme.py
It's Christmas in Heaven...def minmax(test, *args):
res = args[0]
for arg in args[1:]:
if test(arg, res):
res = arg
return res
def lessthan(x, y): return x < y
def grtrthan(x, y): return x > y
print(minmax(lessthan, 4, 2, 1, 5, 6, 3)) # Self-test code
print(minmax(grtrthan, 4, 2, 1, 5, 6, 3))print('I am:', __name__)
def minmax(test, *args):
res = args[0]
for arg in args[1:]:
if test(arg, res):
res = arg
return res
def lessthan(x, y): return x < y
def grtrthan(x, y): return x > y
if __name__ == '__main__':
print(minmax(lessthan, 4, 2, 1, 5, 6, 3)) # Self-test code
print(minmax(grtrthan, 4, 2, 1, 5, 6, 3))c:\code> python minmax2.py
I am: __main__
1
6c:\code>python>>>import minmax2I am: minmax2 >>>minmax2.minmax(minmax2.lessthan, 's', 'p', 'a', 'a')'a'
#!python
"""
File: formats.py (2.X and 3.X)
Various specialized string display formatting utilities.
Test me with canned self-test or command-line arguments.
To do: add parens for negative money, add more features.
"""
def commas(N):
"""
Format positive integer-like N for display with
commas between digit groupings: "xxx,yyy,zzz".
"""
digits = str(N)
assert digits.isdigit()
result = ''
while digits:
digits, last3 = digits[:-3], digits[-3:]
result = (last3 + ',' + result) if result else last3
return result
def money(N, numwidth=0, currency='$'):
"""
Format number N for display with commas, 2 decimal digits,
leading $ and sign, and optional padding: "$ -xxx,yyy.zz".
numwidth=0 for no space padding, currency='' to omit symbol,
and non-ASCII for others (e.g., pound=u'\xA3' or u'\u00A3').
"""
sign = '-' if N < 0 else ''
N = abs(N)
whole = commas(int(N))
fract = ('%.2f' % N)[-2:]
number = '%s%s.%s' % (sign, whole, fract)
return '%s%*s' % (currency, numwidth, number)
if __name__ == '__main__':
def selftest():
tests = 0, 1 # fails: −1, 1.23
tests += 12, 123, 1234, 12345, 123456, 1234567
tests += 2 ** 32, 2 ** 100
for test in tests:
print(commas(test))
print('')
tests = 0, 1, −1, 1.23, 1., 1.2, 3.14159
tests += 12.34, 12.344, 12.345, 12.346
tests += 2 ** 32, (2 ** 32 + .2345)
tests += 1.2345, 1.2, 0.2345
tests += −1.2345, −1.2, −0.2345
tests += −(2 ** 32), −(2**32 + .2345)
tests += (2 ** 100), −(2 ** 100)
for test in tests:
print('%s [%s]' % (money(test, 17), test))
import sys
if len(sys.argv) == 1:
selftest()
else:
print(money(float(sys.argv[1]), int(sys.argv[2])))c:\code>python formats.py0 1 12 123 1,234 12,345 123,456 1,234,567...etc...
C:\code>python formats.py 999999999 0$999,999,999.00 C:\code>python formats.py −999999999 0$-999,999,999.00 C:\code>python formats.py 123456789012345 0$123,456,789,012,345.00 C:\code>python formats.py −123456789012345 25$ −123,456,789,012,345.00 C:\code>python formats.py 123.456 0$123.46 C:\code>python formats.py −123.454 0$-123.45
>>>from formats import money, commas>>>money(123.456)'$123.46' >>>money(-9999999.99, 15)'$ −9,999,999.99' >>>X = 99999999999999999999>>>'%s (%s)' % (commas(X), X)'99,999,999,999,999,999,999 (99999999999999999999)'
u in such string literals in Python
3.3)b in such string literals in Python
2.X)from __future__ import print_function # 2.X
from formats import money
X = 54321.987
print(money(X), money(X, 0, ''))
print(money(X, currency=u'\xA3'), money(X, currency=u'\u00A5'))
print(money(X, currency=b'\xA3'.decode('latin-1')))
print(money(X, currency=u'\u20AC'), money(X, 0, b'\xA4'.decode('iso-8859-15')))
print(money(X, currency=b'\xA4'.decode('latin-1')))$54,321.99 54,321.99 £54,321.99 ¥54,321.99 £54,321.99 €54,321.99 €54,321.99 ¤54,321.99
c:\code>formats_currency.py > tempc:\code>notepad temp
c:\code>chcp 65001# Console matches Pythonc:\code>set PYTHONIOENCODING=utf-8# Python matches consolec:\code>formats_currency.py > temp# Both 3.X and 2.X write UTF-8 textc:\code>type temp# Console displays it properlyc:\code>notepad temp# Notepad recognizes UTF-8 too
c:\code>py −2>>>print u'\xA5' + '1', '%s2' % u'\u00A3'# 2.X: unicode/str mix for ASCII str¥1 £2 c:\code>py −3>>>print(u'\xA5' + '1', '%s2' % u'\u00A3')#3.X: str is Unicode, u'' optional¥1 £2 >>>print('\xA5' + '1', '%s2' % '\u00A3')¥1 £2
>>>import formats>>>help(formats)Help on module formats: NAME formats DESCRIPTION File: formats.py (2.X and 3.X) Various specialized string display formatting utilities. Test me with canned self-test or command-line arguments. To do: add parens for negative money, add more features. FUNCTIONS commas(N) Format positive integer-like N for display with commas between digit groupings: "xxx,yyy,zzz". money(N, numwidth=0, currency='$') Format number N for display with commas, 2 decimal digits, leading $ and sign, and optional padding: "$ -xxx,yyy.zz". numwidth=0 for no space padding, currency='' to omit symbol, and non-ASCII for others (e.g., pound=u'£' or u'£'). FILE c:\code\formats.py
>>>import sys>>>sys.path['', 'c:\\temp', 'C:\\Windows\\system32\\python33.zip',...more deleted...] >>>sys.path.append('C:\\sourcedir')# Extend module search path>>>import string# All imports search the new dir last
>>>sys.path = [r'd:\temp']# Change module search path>>>sys.path.append('c:\\lp5e\\examples')# For this run (process) only>>>sys.path.insert(0, '..')>>>sys.path['..', 'd:\\temp', 'c:\\lp5e\\examples'] >>>import stringTraceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'string'
import modulename as name # And use name, not modulenameimport modulename
name = modulename
del modulename # Don't keep original namefrom modulename import attrname as name # And use name, not attrnameimport reallylongmodulename as name# Use shorter nicknamename.func() from module1 import utility as util1# Can have only 1 "utility"from module2 import utility as util2 util1(); util2()
import dir1.dir2.mod as mod# Only list full path oncemod.func() from dir1.dir2.mod import func as modfunc# Rename to make unique if neededmodfunc()
import newname as oldname
from library import newname as oldname
...and keep happily using oldname until you have time to update all your code...M.name# Qualify object by attributeM.__dict__['name']# Index namespace dictionary manuallysys.modules['M'].name# Index loaded-modules table manuallygetattr(M, 'name')# Call built-in fetch function
#!python
"""
mydir.py: a module that lists the namespaces of other modules
"""
from __future__ import print_function # 2.X compatibility
seplen = 60
sepchr = '-'
def listing(module, verbose=True):
sepline = sepchr * seplen
if verbose:
print(sepline)
print('name:', module.__name__, 'file:', module.__file__)
print(sepline)
count = 0
for attr in sorted(module.__dict__): # Scan namespace keys (or enumerate)
print('%02d) %s' % (count, attr), end = ' ')
if attr.startswith('__'):
print('<built-in name>') # Skip __file__, etc.
else:
print(getattr(module, attr)) # Same as .__dict__[attr]
count += 1
if verbose:
print(sepline)
print(module.__name__, 'has %d names' % count)
print(sepline)
if __name__ == '__main__':
import mydir
listing(mydir) # Self-test code: list myselfc:\code> py −3 mydir.py
------------------------------------------------------------
name: mydir file: c:\code\mydir.py
------------------------------------------------------------
00) __builtins__ <built-in name>
01) __cached__ <built-in name>
02) __doc__ <built-in name>
03) __file__ <built-in name>
04) __initializing__ <built-in name>
05) __loader__ <built-in name>
06) __name__ <built-in name>
07) __package__ <built-in name>
08) listing <function listing at 0x000000000295B488>
09) print_function _Feature((2, 6, 0, 'alpha', 2), (3, 0, 0, 'alpha', 0), 65536)
10) sepchr -
11) seplen 60
------------------------------------------------------------
mydir has 12 names
------------------------------------------------------------>>>import mydir>>>import tkinter>>>mydir.listing(tkinter)------------------------------------------------------------ name: tkinter file: C:\Python33\lib\tkinter\__init__.py ------------------------------------------------------------ 00) ACTIVE active 01) ALL all 02) ANCHOR anchor 03) ARC arc 04) At <function At at 0x0000000002BD41E0>...many more names omitted...156) image_types <function image_types at 0x0000000002BE2378> 157) mainloop <function mainloop at 0x0000000002BCBBF8> 158) sys <module 'sys' (built-in)> 159) wantobjects 1 160) warnings <module 'warnings' from 'C:\\Python33\\lib\\warnings.py'> ------------------------------------------------------------ tkinter has 161 names ------------------------------------------------------------
>>> import 'string'
File "<stdin>", line 1
import "string"
^
SyntaxError: invalid syntaxx = 'string' import x
>>>modname = 'string'>>>exec('import ' + modname)# Run a string of code>>>string# Imported in this namespace<module 'string' from 'C:\\Python33\\lib\\string.py'>
>>>modname = 'string'>>>string = __import__(modname)>>>string<module 'string' from 'C:\\Python33\\lib\\string.py'>
>>>import importlib>>>modname = 'string'>>>string = importlib.import_module(modname)>>>string<module 'string' from 'C:\\Python33\\lib\\string.py'>
# A.pyimport B# Not reloaded when A is!import C# Just an import of an already loaded module: no-ops%python>>> . . . >>>from imp import reload>>>reload(A)
#!python """ reloadall.py: transitively reload nested modules (2.X + 3.X). Call reload_all with one or more imported module objects. """ import types from imp import reload# from required in 3.Xdef status(module): print('reloading ' + module.__name__) def tryreload(module): try: reload(module)# 3.3 (only?) fails on someexcept: print('FAILED: %s' % module) def transitive_reload(module, visited): if not module in visited:# Trap cycles, duplicatesstatus(module)# Reload this moduletryreload(module)# And visit childrenvisited[module] = True for attrobj in module.__dict__.values():# For all attrsif type(attrobj) == types.ModuleType:# Recur if moduletransitive_reload(attrobj, visited) def reload_all(*args): visited = {}# Main entry pointfor arg in args:# For all passed inif type(arg) == types.ModuleType: transitive_reload(arg, visited) def tester(reloader, modname):# Self-test codeimport importlib, sys# Import on tests onlyif len(sys.argv) > 1: modname = sys.argv[1]# command line (or passed)module = importlib.import_module(modname)# Import by name stringreloader(module)# Test passed-in reloaderif __name__ == '__main__': tester(reload_all, 'reloadall')# Test: reload myself?
C:\code>c:\Python33\python reloadall.pyreloading reloadall reloading types c:\code>C:\Python27\python reloadall.pyreloading reloadall reloading types
c:\code> reloadall.py pybench
reloading pybench
reloading timeit
reloading itertools
reloading sys
reloading time
reloading gc
reloading os
reloading errno
reloading ntpath
reloading stat
reloading genericpath
reloading copyreg>>>from reloadall import reload_all>>>import os, tkinter>>>reload_all(os)# Normal usage modereloading os reloading ntpath reloading stat reloading sys reloading genericpath reloading errno reloading copyreg >>>reload_all(tkinter)reloading tkinter reloading _tkinter reloading warnings reloading sys reloading linecache reloading tokenize reloading builtins FAILED: <module 'builtins'> reloading re...etc...reloading os reloading ntpath reloading stat reloading genericpath reloading errno...etc...
import b# File a.pyX = 1 import c# File b.pyY = 2 Z = 3# File c.pyC:\code>py −3>>>import a>>>a.X, a.b.Y, a.b.c.Z(1, 2, 3)# Without stopping Python, change all three files' assignment values and save>>>from imp import reload>>>reload(a)# Built-in reload is top level only<module 'a' from '.\\a.py'> >>>a.X, a.b.Y, a.b.c.Z(111, 2, 3) >>>from reloadall import reload_all>>>reload_all(a)# Normal usage modereloading a reloading b reloading c >>>a.X, a.b.Y, a.b.c.Z# Reloads all nested modules too(111, 222, 333)
""" reloadall2.py: transitively reload nested modules (alternative coding) """ import types from imp import reload# from required in 3.Xfrom reloadall import status, tryreload, tester def transitive_reload(objects, visited): for obj in objects: if type(obj) == types.ModuleType and obj not in visited: status(obj) tryreload(obj)# Reload this, recur to attrsvisited.add(obj) transitive_reload(obj.__dict__.values(), visited) def reload_all(*args): transitive_reload(args, set()) if __name__ == '__main__': tester(reload_all, 'reloadall2')# Test code: reload myself?
""" reloadall3.py: transitively reload nested modules (explicit stack) """ import types from imp import reload# from required in 3.Xfrom reloadall import status, tryreload, tester def transitive_reload(modules, visited): while modules: next = modules.pop()# Delete next item at endstatus(next)# Reload this, push attrstryreload(next) visited.add(next) modules.extend(x for x in next.__dict__.values() if type(x) == types.ModuleType and x not in visited) def reload_all(*modules): transitive_reload(list(modules), set()) if __name__ == '__main__': tester(reload_all, 'reloadall3')# Test code: reload myself?
c:\code>reloadall.pyreloading reloadall reloading types c:\code>reloadall2.pyreloading reloadall2 reloading types c:\code>reloadall3.pyreloading reloadall3 reloading types
c:\code>reloadall.py tkinterreloading tkinter reloading _tkinter reloading tkinter._fix...etc...c:\code>reloadall2.py tkinterreloading tkinter reloading tkinter.constants reloading tkinter._fix...etc...c:\code>reloadall3.py tkinterreloading tkinter reloading sys reloading tkinter.constants...etc...
c:\code>py −2 reloadall.pyreloading reloadall reloading types c:\code>py −2 reloadall2.py Tkinterreloading Tkinter reloading _tkinter reloading FixTk...etc...
C:\code>py −3>>>import reloadall, reloadall2, reloadall3>>>import tkinter>>>reloadall.reload_all(tkinter)# Normal use casereloading tkinter reloading tkinter._fix reloading os...etc...>>>reloadall.tester(reloadall2.reload_all, 'tkinter')# Testing utilityreloading tkinter reloading tkinter._fix reloading os...etc...>>>reloadall.tester(reloadall3.reload_all, 'reloadall3')# Mimic self-test codereloading reloadall3 reloading types
>>>import os>>>res1 = os.popen('reloadall.py tkinter').readlines()>>>res2 = os.popen('reloadall2.py tkinter').readlines()>>>res3 = os.popen('reloadall3.py tkinter').readlines()>>>res1[:3]['reloading tkinter\n', 'reloading sys\n', 'reloading tkinter._fix\n'] >>>res1 == res2, res2 == res3(False, False) >>>set(res1) == set(res2), set(res2) == set(res3)(True, True)
func1()# Error: "func1" not yet assigneddef func1(): print(func2())# OK: "func2" looked up laterfunc1()# Error: "func2" not yet assigneddef func2(): return "Hello" func1()# OK: "func1" and "func2" assigned
# nested1.py
X = 99
def printer(): print(X)# nested2.pyfrom nested1 import X, printer# Copy names outX = 88# Changes my "X" only!printer()# nested1's X is still 99%python nested2.py99
# nested3.pyimport nested1# Get module as a wholenested1.X = 88# OK: change nested1's Xnested1.printer() %python nested3.py88
>>>from module1 import *# Bad: may overwrite my names silently>>>from module2 import *# Worse: no way to tell what we get! >>>from module3 import *>>> . . . >>>func()# Huh???
from module import X# X may not reflect any module reloads!. . . from imp import reload reload(module)# Changes module, but not my namesX# Still references old object
import module# Get module, not names. . . from imp import reload reload(module)# Changes module in placemodule.X# Get current X: reflects module reloads
from module import function function(1, 2, 3)
from imp import reload reload(module)
from imp import reload import module reload(module) function(1, 2, 3)
from imp import reload
import module
reload(module)
from module import function # Or give up and use module.function()
function(1, 2, 3)# recur1.pyX = 1 import recur2# Run recur2 now if it doesn't existY = 2# recur2.pyfrom recur1 import X# OK: "X" already assignedfrom recur1 import Y# Error: "Y" not yet assignedC:\code>py −3>>>import recur1Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".\recur1.py", line 2, in <module> import recur2 File ".\recur2.py", line 2, in <module> from recur1 import Y ImportError: cannot import name Y
import and
attribute qualification (instead of from and direct names), or by running your
froms either inside functions
(instead of at the top level of the module) or near the bottom of
your file to defer their execution.__name__ variable is the string "__main__"?sys.path
different from setting PYTHONPATH
to modify the module search path?__future__
allows us to import from the future, can we also import from the
past?from *
statement form is used. They can still be accessed by an import or the normal from statement form, though. The __all__ list is similar, but the logical
converse; its contents are the only names that
are copied out on a from
*.__name__
variable is the string "__main__",
it means that the file is being executed as a top-level script instead
of being imported from another file in the program. That is, the file
is being used as a program, not a library. This usage mode variable
supports dual-mode code and tests.import statement with exec, or pass the string name in a call to
the __import__ or importlib.import_module.sys.path only
affects one running program (process), and is temporary — the change
goes away when the program ends. PYTHONPATH settings live in the operating
system — they are picked up globally by all your programs on a machine,
and changes to these settings endure after programs exit.countLines(name)
function that reads an input file and counts the number of lines
in it (hint: file.readlines
does most of the work for you, and len does the rest, though you could
count with for and file
iterators to support massive files too).countChars(name)
function that reads an input file and counts the number of
characters in it (hint: file.read returns a single string, which
may be used in similar ways).test(name) function
that calls both counting functions with a given input filename.
Such a filename generally might be passed in, hardcoded, input
with the input built-in
function, or pulled from a command line via the sys.argv list shown in this chapter’s
formats.py and reloadall.py examples; for now, you can
assume it’s a passed-in function argument.mymod functions
should expect a filename string to be passed in. If you type more than
two or three lines per function, you’re working much too hard — use the
hints I just gave!import and attribute references to fetch
your exports. Does your PYTHONPATH
need to include the directory where you created mymod.py? Try running your module on
itself: for example, test("mymod.py"). Note that test opens the file twice; if you’re feeling
ambitious, you may be able to improve this by passing an open file
object into the two count functions (hint: file.seek(0) is a file rewind).from/from *. Test your
mymod module from exercise 1
interactively by using from to load
the exports directly, first by name, then using the from * variant to fetch everything.mymod module that calls the test function automatically only when the
module is run as a script, not when it is imported. The line you add
will probably test the value of __name__ for the string "__main__", as shown in this chapter. Try
running your module from the system command line; then, import the
module and test its functions interactively. Does it still work in
both modes?mymod and tests its functions; then
run myclient from the system
command line. If myclient uses
from to fetch from mymod, will mymod’s functions be accessible from the top
level of myclient? What if it
imports with import instead? Try
coding both variations in myclient
and test interactively by importing myclient and inspecting its __dict__ attribute.import
mypkg.mymod and call its functions. Try to fetch your
counter functions with a from
too.changer in another window, or
suspend the Python interpreter and edit in the same window (on Unix, a
Ctrl-Z key combination usually suspends the current process, and an
fg command later resumes it, though
a text edit window probably works just as well).recur1 raised an error. But if you restart
Python and import recur2
interactively, the error doesn’t occur — test this and see for yourself.
Why do you think it works to import recur2, but not recur1? (Hint: Python stores new modules in
the built-in sys.modules table — a
dictionary — before running their code; later imports fetch the module
from this table first, whether the module is “complete” yet or not.)
Now, try running recur1 as a
top-level script file: python
recur1.py. Do you get the same error that occurs when
recur1 is imported interactively?
Why? (Hint: when modules are run as programs, they aren’t imported, so
this case has the same effect as importing recur2 interactively; recur2 is the first module imported.) What
happens when you run recur2 as a
script? Circular imports are uncommon and rarely this bizarre in
practice. On the other hand, if you can understand why they are a
potential problem, you know a lot about Python’s import semantics.1 As we saw briefly in “Other Ways to Access Globals” in Chapter 17, because a function can access its enclosing
module by going through the sys.modules table like this, it can also be
used to emulate the effect of the global statement. For instance, the effect
of global X; X=0 can be simulated
(albeit with much more typing!) by saying this inside a function:
import sys; glob=sys.modules[__name__];
glob.X=0. Remember, each module gets a __name__ attribute for free; it’s visible as
a global name inside the functions within the module. This trick
provides another way to change both local and global variables of the
same name inside a function.
2 You can preload tools such as mydir.listing and the reloader we’ll meet in
a moment into the interactive namespace by importing them in the file
referenced by the PYTHONSTARTUP
environment variable. Because code in the startup file runs in the
interactive namespace (module __main__), importing common tools in the
startup file can save you some typing. See Appendix A for more details.
Pizza-making robots are kinds of robots, so they possess the usual robot-y properties. In OOP terms, we say they “inherit” properties from the general category of all robots. These common properties need to be implemented only once for the general case and can be reused in part or in full by all types of robots we may build in the future.
Pizza-making robots are really collections of components that work together as a team. For instance, for our robot to be successful, it might need arms to roll dough, motors to maneuver to the oven, and so on. In OOP parlance, our robot is an example of composition; it contains other objects that it activates to do its bidding. Each component might be coded as a class, which defines its own behavior and relationships.
Classes are essentially factories for generating one or more objects. Every time we call a class, we generate a new object with a distinct namespace. Each object generated from a class has access to the class’s attributes and gets a namespace of its own for data that varies per object. This is similar to the per-call state retention of Chapter 17’s closure functions, but is explicit and natural in classes, and is just one of the things that classes do. Classes offer a complete programming solution.
Classes also support the OOP notion of inheritance; we can extend a class by redefining its attributes outside the class itself in new software components coded as subclasses. More generally, classes can build up namespace hierarchies, which define names to be used by objects created from classes in the hierarchy. This supports multiple customizable behaviors more directly than other tools.
By providing special protocol methods, classes can define objects that respond to the sorts of operations we saw at work on built-in types. For instance, objects made with classes can be sliced, concatenated, indexed, and so on. Python provides hooks that classes can use to intercept and implement any built-in type operation.
object.attribute
Find the first occurrence ofattributeby looking inobject, then in all classes above it, from bottom to top and left to right.
Serve as instance factories. Their attributes provide behavior — data and functions — that is inherited by all the instances generated from them (e.g., a function to compute an employee’s salary from pay and hours).
Represent the concrete items in a program’s domain. Their attributes record data that varies per specific object (e.g., an employee’s Social Security number).
I2.w
I2, C1, C2, C3
I1.x and I2.x both find x in C1
and stop because C1 is lower than
C2.I1.y and I2.y both find y in C1
because that’s the only place y
appears.I1.z and I2.z both find z in C2
because C2 is further to the left
than C3.I2.name finds name in I2 without climbing the tree at
all.giveRaise from
bob, by inheritance searchbob to the located
giveRaise function, in the
special self argumentclass statement
generates a new class object.class header line; the left-to-right order
there gives the order in the tree.class C2: ...# Make class objects (ovals)class C3: ... class C1(C2, C3): ...# Linked to superclasses (in this order)I1 = C1()# Make instance objects (rectangles)I2 = C1()# Linked to their classes
class
statement blocks, and not nested inside function def statements there.self.class C2: ...# Make superclass objectsclass C3: ... class C1(C2, C3):# Make and link class C1def setname(self, who):# Assign name: C1.setnameself.name = who# Self is either I1 or I2I1 = C1()# Make two instancesI2 = C1() I1.setname('bob')# Sets I1.name to 'bob'I2.setname('sue')# Sets I2.name to 'sue'print(I1.name)# Prints 'bob'
class C2: ...# Make superclass objectsclass C3: ... class C1(C2, C3): def __init__(self, who):# Set name when constructedself.name = who# Self is either I1 or I2I1 = C1('bob')# Sets I1.name to 'bob'I2 = C1('sue')# Sets I2.name to 'sue'print(I1.name)# Prints 'bob'
class Employee:# General superclassdef computeSalary(self): ...# Common or default behaviorsdef giveRaise(self): ... def promote(self): ... def retire(self): ...
class Engineer(Employee):# Specialized subclassdef computeSalary(self): ...# Something custom here
bob = Employee()# Default behaviorsue = Employee()# Default behaviortom = Engineer()# Custom salary calculator
company = [bob, sue, tom]# A composite objectfor emp in company: print(emp.computeSalary())# Run this object's version: default or custom
def processor(reader, converter, writer):
while True:
data = reader.read()
if not data: break
data = converter(data)
writer.write(data)class Reader:
def read(self): ... # Default behavior and tools
def other(self): ...
class FileReader(Reader):
def read(self): ... # Read from a local file
class SocketReader(Reader):
def read(self): ... # Read from a network socket
...
processor(FileReader(...), Converter, FileWriter(...))
processor(SocketReader(...), Converter, TapeWriter(...))
processor(FtpReader(...), Converter, XmlWriter(...))__init__ method
used for?self by convention. Because method functions
always have this implied subject and object context by default, we say
they are “object-oriented” (i.e., designed to process or change
objects).__init__ method is
coded or inherited in a class, Python calls it automatically each time
an instance of that class is created. It’s known as the constructor
method; it is passed the new instance implicitly, as well as any
arguments passed explicitly to the class name. It’s also the most
commonly used operator overloading method. If no __init__ method is present, instances simply
begin life as empty namespaces.__init__ constructor method. The new
instance remembers the class it was created from for inheritance
purposes.class statement; like function definitions,
these statements normally run when the enclosing module file is
imported (more on this in the next chapter).class statement,
after the new class’s name. The left-to-right order in which the
classes are listed in the parentheses gives the left-to-right
inheritance search order in the class tree.1 In other literature and circles, you may also occasionally see the terms base classes and derived classes used to describe superclasses and subclasses, respectively. Python people and this book tend to use the latter terms.
2 If you’ve ever used C++ or Java, you’ll recognize that
Python’s self is the same as the
this pointer, but
self is always explicit in both headers and
bodies of Python methods to make attribute accesses more obvious: a
name has fewer possible meanings.
3 The company list in this
example could be a database if stored in a file with Python object
pickling, introduced in Chapter 9, to make the
employees persistent. Python also comes with a module named
shelve, which allows the
pickled representation of class instances to be stored in an
access-by-key filesystem; we’ll deploy it in Chapter 28.
class statement
creates a class object and assigns it a
name. Just like the function def statement, the Python class statement is an
executable statement. When reached and run, it
generates a new class object and assigns it to the name in the
class header. Also, like defs, class statements typically run when the
files they are coded in are first imported.class statements make class attributes. Just like
in module files, top-level assignments within a class statement (not nested in a def) generate attributes in a class
object. Technically, the class
statement defines a local scope that morphs
into the attribute namespace of the class object, just like a
module’s global scope. After running a class statement, class attributes are
accessed by name qualification:
object.name.def statements nested inside a class generate
methods, which process instances.self in methods make per-instance attributes.
Inside a class’s method functions, the first argument (called
self by convention) references
the instance object being processed; assignments to attributes of
self create or change data in the
instance, not the class.>>>class FirstClass:# Define a class objectdef setdata(self, value):# Define class's methodsself.data = value# self is the instancedef display(self):print(self.data)# self.data: per instance
>>>x = FirstClass()# Make two instances>>>y = FirstClass()# Each is a new namespace
>>>x.setdata("King Arthur")# Call methods: self is x>>>y.setdata(3.14159)# Runs: FirstClass.setdata(y, 3.14159)
>>>x.display()# self.data differs in each instanceKing Arthur >>>y.display()# Runs: FirstClass.display(y)3.14159
>>>x.data = "New value"# Can get/set attributes>>>x.display()# Outside the class tooNew value
>>>x.anothername = "spam"# Can set new attributes here too!
class header. To make a class inherit attributes from another class, just
list the other class in parentheses in the new class statement’s header line. The class
that inherits is usually called a subclass, and
the class that is inherited from is its
superclass.object.attribute
reference invokes a new, independent
search. Python performs an independent search of the class
tree for each attribute fetch expression. This includes references to
instances and classes made outside class statements (e.g., X.attr), as well
as references to attributes of the self instance argument in a class’s method
functions. Each self.attr
expression in a method invokes a new search for
attr in self and above.>>>class SecondClass(FirstClass):# Inherits setdatadef display(self):# Changes displayprint('Current value = "%s"' % self.data)
>>>z = SecondClass()>>>z.setdata(42)# Finds setdata in FirstClass>>>z.display()# Finds overridden method in SecondClassCurrent value = "42"
>>>x.display()# x is still a FirstClass instance (old message)New value
from modulename import FirstClass# Copy name into my scopeclass SecondClass(FirstClass):# Use class name directlydef display(self): ...
import modulename# Access the whole moduleclass SecondClass(modulename.FirstClass):# Qualify to referencedef display(self): ...
# food.pyvar = 1# food.vardef func(): ...# food.funcclass spam: ...# food.spamclass ham: ...# food.hamclass eggs: ...# food.eggs
class person: ...
import person# Import modulex = person.person()# Class within module
from person import person# Get class from modulex = person()# Use class name
import person# Lowercase for modulesx = person.Person()# Uppercase for classes
__X__) are special
hooks. In Python classes we implement operator overloading by
providing specially named methods to intercept operations. The Python
language defines a fixed and unchangeable mapping from each of these
operations to a specially named method.__add__ method, that method is called
whenever the object appears in a +
expression. The method’s return value becomes the result of the
corresponding expression.__add__,
for example, + expressions raise
exceptions.object does provide defaults for
some __X__ methods, but not for many, and not for
most commonly used operations.__init__ is run when a new
instance object is created: self
is the new ThirdClass
object.1__add__ is run when a
ThirdClass instance appears in a
+ expression.__str__ is run when an
object is printed (technically, when it’s converted to its print
string by the str built-in
function or its Python internals equivalent).>>>class ThirdClass(SecondClass):# Inherit from SecondClassdef __init__(self, value):# On "ThirdClass(value)"self.data = valuedef __add__(self, other):# On "self + other"return ThirdClass(self.data + other)def __str__(self):# On "print(self)", "str()"return '[ThirdClass: %s]' % self.datadef mul(self, other):# In-place change: namedself.data *= other>>>a = ThirdClass('abc')# __init__ called>>>a.display()# Inherited method calledCurrent value = "abc" >>>print(a)# __str__: returns display string[ThirdClass: abc] >>>b = a + 'xyz'# __add__: makes a new instance>>>b.display()# b has all ThirdClass methodsCurrent value = "abcxyz" >>>print(b)# __str__: returns display string[ThirdClass: abcxyz] >>>a.mul(3)# mul: changes instance in place>>>print(a)[ThirdClass: abcabcabc]
>>>class rec: pass# Empty namespace object
>>>rec.name = 'Bob'# Just objects with attributes>>>rec.age = 40
>>>print(rec.name)# Like a C struct or a recordBob
>>>x = rec()# Instances inherit class names>>>y = rec()
>>>x.name, y.name# name is stored on the class only('Bob', 'Bob')
>>>x.name = 'Sue'# But assignment changes x only>>>rec.name, x.name, y.name('Bob', 'Sue', 'Bob')
>>>list(rec.__dict__.keys())['age', '__module__', '__qualname__', '__weakref__', 'name', '__dict__', '__doc__'] >>>list(name for name in rec.__dict__ if not name.startswith('__'))['age', 'name'] >>>list(x.__dict__.keys())['name'] >>>list(y.__dict__.keys())# list() not required in Python 2.X[]
>>>x.name, x.__dict__['name']# Attributes present here are dict keys('Sue', 'Sue') >>>x.age# But attribute fetch checks classes too40 >>>x.__dict__['age']# Indexing dict does not do inheritanceKeyError: 'age'
>>>x.__class__# Instance to class link<class '__main__.rec'>
>>>rec.__bases__# Class to superclasses link, () in 2.X(<class 'object'>,)
>>>def uppername(obj):return obj.name.upper()# Still needs a self argument (obj)
>>>uppername(x)# Call as a simple function'SUE'
>>>rec.method = uppername# Now it's a class's method!>>>x.method()# Run method to process x'SUE' >>>y.method()# Same, but pass y to self'BOB' >>>rec.method(x)# Can call through instance or class'SUE'
>>>rec = ('Bob', 40.5, ['dev', 'mgr'])# Tuple-based record>>>print(rec[0])Bob >>>rec = {}>>>rec['name'] = 'Bob'# Dictionary-based record>>>rec['age'] = 40.5# Or {...}, dict(n=v), etc.>>>rec['jobs'] = ['dev', 'mgr']>>> >>>print(rec['name'])Bob
>>>class rec: pass>>>rec.name = 'Bob'# Class-based record>>>rec.age = 40.5>>>rec.jobs = ['dev', 'mgr']>>> >>>print(rec.name)Bob
>>>class rec: pass>>>pers1 = rec()# Instance-based records>>>pers1.name = 'Bob'>>>pers1.jobs = ['dev', 'mgr']>>>pers1.age = 40.5>>> >>>pers2 = rec()>>>pers2.name = 'Sue'>>>pers2.jobs = ['dev', 'cto']>>> >>>pers1.name, pers2.name('Bob', 'Sue')
>>>class Person:def __init__(self, name, jobs, age=None):# class = data + logicself.name = nameself.jobs = jobsself.age = agedef info(self):return (self.name, self.jobs)>>>rec1 = Person('Bob', ['dev', 'mgr'], 40.5)# Construction calls>>>rec2 = Person('Sue', ['dev', 'cto'])>>> >>>rec1.jobs, rec2.info()# Attributes + methods(['dev', 'mgr'], ('Sue', ['dev', 'cto']))
>>>rec = dict(name='Bob', age=40.5, jobs=['dev', 'mgr'])# Dictionaries>>>rec = {'name': 'Bob', 'age': 40.5, 'jobs': ['dev', 'mgr']}>>>rec = Rec('Bob', 40.5, ['dev', 'mgr'])# Named tuples
self mean in a
Python class?class statements; instances are created by
calling a class as though it were a function.class statement — each name
assigned in the class statement
block becomes an attribute of the class object (technically, the
class statement’s local scope
morphs into the class object’s attribute namespace, much like a
module). Class attributes can also be created, though, by assigning
attributes to the class anywhere a reference to the class object
exists — even outside the class
statement.class
statement, by assigning attributes to the self argument (which is always the implied
instance). Again, though, they may be created by assignment anywhere a
reference to the instance appears, even outside the class statement. Normally, all instance
attributes are initialized in the __init__ constructor method; that way, later
method calls can assume the attributes already exist.self is the name commonly given to the first
(leftmost) argument in a class’s method function; Python automatically fills it in with
the instance object that is the implied subject of the method call. This argument need not
be called self (though this is a very strong
convention); its position is what is significant. (Ex-C++ or Java programmers might prefer to call it this because in those languages that name reflects the same idea; in Python,
though, this argument must always be explicit.)__init__ constructor
method is the most commonly used; almost every class uses this method
to set initial values for instance attributes and perform other
startup tasks.self argument in method functions and
the __init__ constructor method are the two
cornerstones of OOP code in Python; if you get these, you should be able to read the text
of most OOP Python code — apart from these, it’s largely just packages of functions. The
inheritance search matters too, of course, but self
represents the automatic object argument, and __init__ is
widespread.1 Not to be confused with the __init__.py files in module packages! The method here is a class constructor function used to initialize the newly created instance, not a module package. See Chapter 24 for more details.
2 In fact, this is one of the reasons the self argument must
always be explicit in Python methods — because methods can be created as
simple functions independent of a class, they need to make the implied
instance argument explicit. They can be called as either functions or
methods, and Python can neither guess nor assume that a simple
function might eventually become a class’s method. The main reason for
the explicit self argument, though,
is to make the meanings of names more obvious: names not referenced
through self are simple variables
mapped to scopes, while names referenced through self with attribute notation are obviously
instance attributes.
Person — a class that creates and
processes information about peopleManager — a customization of
Person that modifies inherited
behavior# File person.py (start)class Person:# Start a class
# Add record field initializationclass Person: def __init__(self, name, job, pay):# Constructor takes three argumentsself.name = name# Fill out fields when createdself.job = job# self is the new instance objectself.pay = pay
# Add defaults for constructor argumentsclass Person: def __init__(self, name, job=None, pay=0):# Normal function argsself.name = name self.job = job self.pay = pay
# Add incremental self-test codeclass Person: def __init__(self, name, job=None, pay=0): self.name = name self.job = job self.pay = pay bob = Person('Bob Smith')# Test the classsue = Person('Sue Jones', job='dev', pay=100000)# Runs __init__ automaticallyprint(bob.name, bob.pay)# Fetch attached attributesprint(sue.name, sue.pay)# sue's and bob's attrs differ
C:\code> person.py
Bob Smith 0
Sue Jones 100000# Allow this file to be imported as well as run/testedclass Person: def __init__(self, name, job=None, pay=0): self.name = name self.job = job self.pay = pay if __name__ == '__main__':# When run for testing only# self-test code bob = Person('Bob Smith') sue = Person('Sue Jones', job='dev', pay=100000) print(bob.name, bob.pay) print(sue.name, sue.pay)
C:\code>person.pyBob Smith 0 Sue Jones 100000 C:\code>pythonPython 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) ... >>>import person>>>
>>>name = 'Bob Smith'# Simple string, outside class>>>name.split()# Extract last name['Bob', 'Smith'] >>>name.split()[-1]# Or [1], if always just two parts'Smith'
>>>pay = 100000# Simple variable, outside class>>>pay *= 1.10# Give a 10% raise>>>print('%.2f' % pay)# Or: pay = pay * 1.10, if you like to type110000.00# Or: pay = pay + (pay * .10), if you _really_ do!
# Process embedded built-in types: strings, mutabilityclass Person: def __init__(self, name, job=None, pay=0): self.name = name self.job = job self.pay = pay if __name__ == '__main__': bob = Person('Bob Smith') sue = Person('Sue Jones', job='dev', pay=100000) print(bob.name, bob.pay) print(sue.name, sue.pay) print(bob.name.split()[-1])# Extract object's last namesue.pay *= 1.10# Give this object a raiseprint('%.2f' % sue.pay)
Bob Smith 0 Sue Jones 100000 Smith 110000.00
# Add methods to encapsulate operations for maintainabilityclass Person: def __init__(self, name, job=None, pay=0): self.name = name self.job = job self.pay = pay def lastName(self):# Behavior methodsreturn self.name.split()[-1]# self is implied subjectdef giveRaise(self, percent): self.pay = int(self.pay * (1 + percent))# Must change here onlyif __name__ == '__main__': bob = Person('Bob Smith') sue = Person('Sue Jones', job='dev', pay=100000) print(bob.name, bob.pay) print(sue.name, sue.pay) print(bob.lastName(), sue.lastName())# Use the new methodssue.giveRaise(.10)# instead of hardcodingprint(sue.pay)
Bob Smith 0 Sue Jones 100000 Smith Jones 110000
bob.lastName(), bob is the implied subject passed to
self.sue.lastName(), sue goes to self instead. @rangetest(percent=(0.0, 1.0)) # Use decorator to validate
def giveRaise(self, percent):
self.pay = int(self.pay * (1 + percent))Bob Smith 0 Sue Jones 100000 Smith Jones <__main__.Person object at 0x00000000029A0668>
# Add __repr__ overload method for printing objectsclass Person: def __init__(self, name, job=None, pay=0): self.name = name self.job = job self.pay = pay def lastName(self): return self.name.split()[-1] def giveRaise(self, percent): self.pay = int(self.pay * (1 + percent)) def __repr__(self):# Added methodreturn '[Person: %s, %s]' % (self.name, self.pay)# String to printif __name__ == '__main__': bob = Person('Bob Smith') sue = Person('Sue Jones', job='dev', pay=100000) print(bob) print(sue) print(bob.lastName(), sue.lastName()) sue.giveRaise(.10) print(sue)
[Person: Bob Smith, 0] [Person: Sue Jones, 100000] Smith Jones [Person: Sue Jones, 110000]
class Manager(Person): # Define a subclass of Personclass Manager(Person):# Inherit Person attrsdef giveRaise(self, percent, bonus=.10):# Redefine to customize
class Manager(Person):
def giveRaise(self, percent, bonus=.10):
self.pay = int(self.pay * (1 + percent + bonus)) # Bad: cut and pasteclass Manager(Person):
def giveRaise(self, percent, bonus=.10):
Person.giveRaise(self, percent + bonus) # Good: augment originalinstance.method(args...)
class.method(instance,args...)
# Add customization of one behavior in a subclassclass Person: def __init__(self, name, job=None, pay=0): self.name = name self.job = job self.pay = pay def lastName(self): return self.name.split()[-1] def giveRaise(self, percent): self.pay = int(self.pay * (1 + percent)) def __repr__(self): return '[Person: %s, %s]' % (self.name, self.pay) class Manager(Person): def giveRaise(self, percent, bonus=.10):# Redefine at this levelPerson.giveRaise(self, percent + bonus)# Call Person's versionif __name__ == '__main__': bob = Person('Bob Smith') sue = Person('Sue Jones', job='dev', pay=100000) print(bob) print(sue) print(bob.lastName(), sue.lastName()) sue.giveRaise(.10) print(sue) tom = Manager('Tom Jones', 'mgr', 50000)# Make a Manager: __init__tom.giveRaise(.10)# Runs custom versionprint(tom.lastName())# Runs inherited methodprint(tom)# Runs inherited __repr__
[Person: Bob Smith, 0] [Person: Sue Jones, 100000] Smith Jones [Person: Sue Jones, 110000] Jones [Person: Tom Jones, 60000]
if __name__ == '__main__':
...
print('--All three--')
for obj in (bob, sue, tom): # Process objects generically
obj.giveRaise(.10) # Run this object's giveRaise
print(obj) # Run the common __repr__[Person: Bob Smith, 0] [Person: Sue Jones, 100000] Smith Jones [Person: Sue Jones, 110000] Jones [Person: Tom Jones, 60000] --All three-- [Person: Bob Smith, 0] [Person: Sue Jones, 121000] [Person: Tom Jones, 72000]
class Person:
def lastName(self): ...
def giveRaise(self): ...
def __repr__(self): ...
class Manager(Person): # Inherit
def giveRaise(self, ...): ... # Customize
def someThingElse(self, ...): ... # Extend
tom = Manager()
tom.lastName() # Inherited verbatim
tom.giveRaise() # Customized version
tom.someThingElse() # Extension here
print(tom) # Inherited overload methodManager from scratch
as new, independent code, we would have had to reimplement all the
behaviors in Person that are the
same for Managers.Person class in place
for the requirements of Manager’s
giveRaise, doing so would
probably break the places where we still need the original Person behavior.Person class in its entirety,
renamed the copy to Manager, and
changed its giveRaise, doing so
would introduce code redundancy that would double our work in the
future — changes made to Person in
the future would not be picked up automatically, but would have to
be manually propagated to Manager’s code. As usual, the
cut-and-paste approach may seem quick now, but it doubles your work
in the future.# File person.py# Add customization of constructor in a subclassclass Person: def __init__(self, name, job=None, pay=0): self.name = name self.job = job self.pay = pay def lastName(self): return self.name.split()[-1] def giveRaise(self, percent): self.pay = int(self.pay * (1 + percent)) def __repr__(self): return '[Person: %s, %s]' % (self.name, self.pay) class Manager(Person): def __init__(self, name, pay):# Redefine constructorPerson.__init__(self, name, 'mgr', pay)# Run original with 'mgr'def giveRaise(self, percent, bonus=.10): Person.giveRaise(self, percent + bonus) if __name__ == '__main__': bob = Person('Bob Smith') sue = Person('Sue Jones', job='dev', pay=100000) print(bob) print(sue) print(bob.lastName(), sue.lastName()) sue.giveRaise(.10) print(sue) tom = Manager('Tom Jones', 50000)# Job name not needed:tom.giveRaise(.10)# Implied/set by classprint(tom.lastName()) print(tom)
[Person: Bob Smith, 0] [Person: Sue Jones, 100000] Smith Jones [Person: Sue Jones, 110000] Jones [Person: Tom Jones, 60000]
# File person-composite.py# Embedding-based Manager alternativeclass Person:...same...class Manager: def __init__(self, name, pay): self.person = Person(name, 'mgr', pay)# Embed a Person objectdef giveRaise(self, percent, bonus=.10): self.person.giveRaise(percent + bonus)# Intercept and delegatedef __getattr__(self, attr): return getattr(self.person, attr)# Delegate all other attrsdef __repr__(self): return str(self.person)# Must overload again (in 3.X)if __name__ == '__main__':...same...
# File person-department.py# Aggregate embedded objects into a compositeclass Person:...same...class Manager(Person):...same...class Department: def __init__(self, *args): self.members = list(args) def addMember(self, person): self.members.append(person) def giveRaises(self, percent): for person in self.members: person.giveRaise(percent) def showAll(self): for person in self.members: print(person) if __name__ == '__main__': bob = Person('Bob Smith') sue = Person('Sue Jones', job='dev', pay=100000) tom = Manager('Tom Jones', 50000) development = Department(bob, sue)# Embed objects in a compositedevelopment.addMember(tom) development.giveRaises(.10)# Runs embedded objects' giveRaisedevelopment.showAll()# Runs embedded objects' __repr__
[Person: Bob Smith, 0] [Person: Sue Jones, 110000] [Person: Tom Jones, 60000]
tom the Manager, the display labels him as a
Person. That’s not technically
incorrect, since Manager is a kind
of customized and specialized Person. Still, it would be more accurate to
display an object with the most specific (that is,
lowest) class possible: the one an object is made
from.__repr__, and that might not
account for future goals. For example, we can’t yet verify that
tom’s job name has been set to
mgr correctly by Manager’s constructor, because the __repr__ we coded for Person does not print this field. Worse, if
we ever expand or otherwise change the set of attributes assigned to
our objects in __init__, we’ll have
to remember to also update __repr__
for new names to be displayed, or it will become out of sync over
time.instance.__class__ attribute provides a link from an instance to the class from
which it was created. Classes in turn have a __name__, just like modules, and a
__bases__ sequence that provides
access to superclasses. We can use these here to print the name of
the class from which an instance is made rather than one we’ve
hardcoded.object.__dict__ attribute provides a dictionary with one key/value pair for
every attribute attached to a namespace object (including modules,
classes, and instances). Because it is a dictionary, we can fetch
its keys list, index by key, iterate over its keys, and so on, to
process all attributes generically. We can use this here to print
every attribute in any instance, not just those we hardcode in
custom displays, much as we did in Chapter 25’s module tools.>>>from person import Person>>>bob = Person('Bob Smith')>>>bob# Show bob's __repr__ (not __str__)[Person: Bob Smith, 0] >>>print(bob)# Ditto: print => __str__ or __repr__[Person: Bob Smith, 0] >>>bob.__class__# Show bob's class and its name<class 'person.Person'> >>>bob.__class__.__name__'Person' >>>list(bob.__dict__.keys())# Attributes are really dict keys['pay', 'job', 'name']# Use list to force list in 3.X>>>for key in bob.__dict__:print(key, '=>', bob.__dict__[key])# Index manuallypay => 0 job => None name => Bob Smith >>>for key in bob.__dict__:print(key, '=>', getattr(bob, key))# obj.attr, but attr is a varpay => 0 job => None name => Bob Smith
# File classtools.py (new)"Assorted class utilities and tools" class AttrDisplay: """ Provides an inheritable display overload method that shows instances with their class names and a name=value pair for each attribute stored on the instance itself (but not attrs inherited from its classes). Can be mixed into any class, and will work on any instance. """ def gatherAttrs(self): attrs = [] for key in sorted(self.__dict__): attrs.append('%s=%s' % (key, getattr(self, key))) return ', '.join(attrs) def __repr__(self): return '[%s: %s]' % (self.__class__.__name__, self.gatherAttrs()) if __name__ == '__main__': class TopTest(AttrDisplay): count = 0 def __init__(self): self.attr1 = TopTest.count self.attr2 = TopTest.count+1 TopTest.count += 2 class SubTest(TopTest): pass X, Y = TopTest(), SubTest()# Make two instancesprint(X)# Show all instance attrsprint(Y)# Show lowest class name
C:\code> classtools.py
[TopTest: attr1=0, attr2=1]
[SubTest: attr1=2, attr2=3]>>>from person import Person# 2.X: keys is list, dir shows less>>>bob = Person('Bob Smith')>>>bob.__dict__.keys()# Instance attrs only['pay', 'job', 'name'] >>>dir(bob)# Plus inherited attrs in classes['__doc__', '__init__', '__module__', '__repr__', 'giveRaise', 'job', 'lastName', 'name', 'pay']
>>>list(bob.__dict__.keys())# 3.X keys is a view, not a list['name', 'job', 'pay'] >>>dir(bob)# 3.X includes class type methods['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__',...more omitted: 31 attrs...'__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'giveRaise', 'job', 'lastName', 'name', 'pay']
>>>len(dir(bob))31 >>>list(name for name in dir(bob) if not name.startswith('__'))['giveRaise', 'job', 'lastName', 'name', 'pay']
class TopTest(AttrDisplay):
....
def gatherAttrs(self): # Replaces method in AttrDisplay!
return 'Spam'# File classtools.py (new)...as listed earlier...# File person.py (final)""" Record and process information about people. Run this file directly to test its classes. """ from classtools import AttrDisplay# Use generic display toolclass Person(AttrDisplay):# Mix in a repr at this level""" Create and process person records """ def __init__(self, name, job=None, pay=0): self.name = name self.job = job self.pay = pay def lastName(self):# Assumes last is lastreturn self.name.split()[-1] def giveRaise(self, percent):# Percent must be 0..1self.pay = int(self.pay * (1 + percent)) class Manager(Person): """ A customized Person with special requirements """ def __init__(self, name, pay): Person.__init__(self, name, 'mgr', pay)# Job name is implieddef giveRaise(self, percent, bonus=.10): Person.giveRaise(self, percent + bonus) if __name__ == '__main__': bob = Person('Bob Smith') sue = Person('Sue Jones', job='dev', pay=100000) print(bob) print(sue) print(bob.lastName(), sue.lastName()) sue.giveRaise(.10) print(sue) tom = Manager('Tom Jones', 50000) tom.giveRaise(.10) print(tom.lastName()) print(tom)
C:\code> person.py
[Person: job=None, name=Bob Smith, pay=0]
[Person: job=dev, name=Sue Jones, pay=100000]
Smith Jones
[Person: job=dev, name=Sue Jones, pay=110000]
Jones
[Manager: job=mgr, name=Tom Jones, pay=60000]import person# Load class with importbob = person.Person(...)# Go through module namefrom person import Person# Load class with frombob = Person(...)# Use name directly
# File makedb.py: store Person objects on a shelve databasefrom person import Person, Manager# Load our classesbob = Person('Bob Smith')# Re-create objects to be storedsue = Person('Sue Jones', job='dev', pay=100000) tom = Manager('Tom Jones', 50000) import shelve db = shelve.open('persondb')# Filename where objects are storedfor obj in (bob, sue, tom):# Use object's name attr as keydb[obj.name] = obj# Store object on shelve by keydb.close()# Close after making changes
C:\code> makedb.py>>>import glob>>>glob.glob('person*')['person-composite.py', 'person-department.py', 'person.py', 'person.pyc', 'persondb.bak', 'persondb.dat', 'persondb.dir'] >>>print(open('persondb.dir').read())'Sue Jones', (512, 92) 'Tom Jones', (1024, 91) 'Bob Smith', (0, 80) >>>print(open('persondb.dat','rb').read())b'\x80\x03cperson\nPerson\nq\x00)\x81q\x01}q\x02(X\x03\x00\x00\x00jobq\x03NX\x03\x00...more omitted...
>>>import shelve>>>db = shelve.open('persondb')# Reopen the shelve>>>len(db)# Three 'records' stored3 >>>list(db.keys())# keys is the index['Sue Jones', 'Tom Jones', 'Bob Smith']# list() to make a list in 3.X>>>bob = db['Bob Smith']# Fetch bob by key>>>bob# Runs __repr__ from AttrDisplay[Person: job=None, name=Bob Smith, pay=0] >>>bob.lastName()# Runs lastName from Person'Smith' >>>for key in db:# Iterate, fetch, printprint(key, '=>', db[key])Sue Jones => [Person: job=dev, name=Sue Jones, pay=100000] Tom Jones => [Manager: job=mgr, name=Tom Jones, pay=50000] Bob Smith => [Person: job=None, name=Bob Smith, pay=0] >>>for key in sorted(db):print(key, '=>', db[key])# Iterate by sorted keysBob Smith => [Person: job=None, name=Bob Smith, pay=0] Sue Jones => [Person: job=dev, name=Sue Jones, pay=100000] Tom Jones => [Manager: job=mgr, name=Tom Jones, pay=50000]
sys.path module
search path (and shouldn’t live in the topmost script files’ module
__main__ unless they’re always in that module when used). Because of
this external module file requirement, some applications choose to
pickle simpler objects such as dictionaries or lists, especially if
they are to be transferred across the Internet.# File updatedb.py: update Person object on databaseimport shelve db = shelve.open('persondb')# Reopen shelve with same filenamefor key in sorted(db):# Iterate to display database objectsprint(key, '\t=>', db[key])# Prints with custom formatsue = db['Sue Jones']# Index by key to fetchsue.giveRaise(.10)# Update in memory using class's methoddb['Sue Jones'] = sue# Assign to key to update in shelvedb.close()# Close after making changes
C:\code>updatedb.pyBob Smith => [Person: job=None, name=Bob Smith, pay=0] Sue Jones => [Person: job=dev, name=Sue Jones, pay=100000] Tom Jones => [Manager: job=mgr, name=Tom Jones, pay=50000] C:\code>updatedb.pyBob Smith => [Person: job=None, name=Bob Smith, pay=0] Sue Jones => [Person: job=dev, name=Sue Jones, pay=110000] Tom Jones => [Manager: job=mgr, name=Tom Jones, pay=50000] C:\code>updatedb.pyBob Smith => [Person: job=None, name=Bob Smith, pay=0] Sue Jones => [Person: job=dev, name=Sue Jones, pay=121000] Tom Jones => [Manager: job=mgr, name=Tom Jones, pay=50000] C:\code>updatedb.pyBob Smith => [Person: job=None, name=Bob Smith, pay=0] Sue Jones => [Person: job=dev, name=Sue Jones, pay=133100] Tom Jones => [Manager: job=mgr, name=Tom Jones, pay=50000]
C:\code>python>>>import shelve>>>db = shelve.open('persondb')# Reopen database>>>rec = db['Sue Jones']# Fetch object by key>>>rec[Person: job=dev, name=Sue Jones, pay=146410] >>>rec.lastName()'Jones' >>>rec.pay146410
As is, we can only process our database with the interactive prompt’s command-based interface, and scripts. We could also work on expanding our object database’s usability by adding a desktop graphical user interface for browsing and updating its records. GUIs can be built portably with either Python’stkinter(Tkinterin 2.X) standard library support, or third-party toolkits such as WxPython and PyQt.tkinterships with Python, lets you build simple GUIs quickly, and is ideal for learning GUI programming techniques; WxPython and PyQt tend to be more complex to use but often produce higher-grade GUIs in the end.
Although GUIs are convenient and fast, the Web is hard to beat in terms of accessibility. We might also implement a website for browsing and updating records, instead of or in addition to GUIs and the interactive prompt. Websites can be constructed with either basic CGI scripting tools that come with Python, or full-featured third-party web frameworks such as Django, TurboGears, Pylons, web2Py, Zope, or Google’s App Engine. On the Web, your data can still be stored in a shelve, pickle file, or other Python-based medium; the scripts that process it are simply run automatically on a server in response to requests from web browsers and other clients, and they produce HTML to interact with a user, either directly or by interfacing with framework APIs. Rich Internet application (RIA) systems such as Silverlight and pyjamas also attempt to combine GUI-like interactivity with web-based deployment.
Although web clients can often parse information in the replies from websites (a technique colorfully known as “screen scraping”), we might go further and provide a more direct way to fetch records on the Web via a web services interface such as SOAP or XML-RPC calls — APIs supported by either Python itself or the third-party open source domain, which generally map data to and from XML format for transmission. To Python scripts, such APIs return data more directly than text embedded in the HTML of a reply page.
If our database becomes higher-volume or critical, we might eventually move it from shelves to a more full-featured storage mechanism such as the open source ZODB object-oriented database system (OODB), or a more traditional SQL-based relational database system such as MySQL, Oracle, or PostgreSQL. Python itself comes with the in-process SQLite database system built-in, but other open source options are freely available on the Web. ZODB, for example, is similar to Python’sshelvebut addresses many of its limitations, better supporting larger databases, concurrent updates, transaction processing, and automatic write-through on in-memory changes (shelves can cache objects and flush to disk at close time with theirwritebackoption, but this has limitations: see other resources). SQL-based systems like MySQL offer enterprise-level tools for database storage and may be directly used from a Python script. As we saw in Chapter 9, MongoDB offers an alternative approach that stores JSON documents, which closely parallel Python dictionaries and lists, and are language neutral, unlikepickledata.
If we do migrate to a relational database system for storage, we don’t have to sacrifice Python’s OOP tools. Object-relational mappers (ORMs) like SQLObject and SQLAlchemy can automatically map relational tables and rows to and from Python classes and instances, such that we can process the stored data using normal Python class syntax. This approach provides an alternative to OODBs likeshelveand ZODB and leverages the power of both relational databases and Python’s class model.
Manager
object from the shelve and print it, where does the display format
logic come from?Person object
from a shelve without importing its module, how does the object know
that it has a giveRaise method that
we can call?__dict__ that allow objects to be processed
generically than to write more custom code for each type of
class?Manager ultimately inherits its __repr__ printing method from AttrDisplay in the separate classtools module and two levels up in the
class tree. Manager doesn’t have
one itself, so the inheritance search climbs to its Person superclass; because there is no
__repr__ there either, the search
climbs higher and finds it in AttrDisplay. The class names listed in
parentheses in a class statement’s
header line provide the links to higher superclasses.pickle
module they use) automatically relink an instance to the class it was
created from when that instance is later loaded back into memory.
Python reimports the class from its module internally, creates an
instance with its stored attributes, and sets the instance’s __class__ link to point to its original
class. This way, loaded instances automatically obtain all their
original methods (like lastName,
giveRaise, and __repr__), even if we have not imported the
instance’s class into our scope.__repr__ print method, for example,
need not be updated each time a new attribute is added to instances in
an __init__ constructor. In
addition, a generic print method
inherited by all classes appears and need be modified in only one
place — changes in the generic version are picked up by all classes that
inherit from the generic class. Again, eliminating code
redundancy cuts future development effort; that’s
one of the primary assets classes bring to the table.Manager
specialization of Person).
Composition is well suited to scenarios where multiple objects are
aggregated into a whole and directed by a controller layer class.
Inheritance passes calls up to reuse, and
composition passes down to delegate. Inheritance
and composition are not mutually exclusive; often, the objects
embedded in a controller are themselves customizations based upon
inheritance.lastName method would need to be
updated for the new name format; the Person constructor would have to change the
job default to an empty list; and the Manager class would probably need to pass
along a job list in its constructor instead of a single string
(self-test code would change as well, of course). The good news is
that these changes would need to be made in just one place — in our
classes, where such details are encapsulated. The database scripts
should work as is, as shelves support arbitrarily nested data.name, address, birthday, phone, email, and so on for a contacts database,
and methods appropriate for this purpose. A method named sendmail, for example, might use Python’s
standard library smtplib module to
send an email to one of the contacts automatically when called (see
Python’s manuals or application-level books for more details on such
tools). The AttrDisplay tool we
wrote here could be used verbatim to print your objects, because it is
intentionally generic. Most of the shelve database code here can be
used to store your objects, too, with minor changes.1 And no offense to any managers in the audience, of course. I once taught a Python class in New Jersey, and nobody laughed at this joke, among others. The organizers later told me it was a group of managers evaluating Python.
2 Yes, we use “shelve” as a noun in Python, much to the chagrin of a variety of editors I’ve worked with over the years, both electronic and human.
classname(superclass,...):# Assign to nameattr= value# Shared class datadefmethod(self,...):# Methodsself.attr= value# Per-instance data
class
statements are local scopes where names created by nested
assignments live.class statement become attributes in a
class object.>>>class SharedData:spam = 42# Generates a class data attribute>>>x = SharedData()# Make two instances>>>y = SharedData()>>>x.spam, y.spam# They inherit and share 'spam' (a.k.a. SharedData.spam)(42, 42)
>>>SharedData.spam = 99>>>x.spam, y.spam, SharedData.spam(99, 99, 99)
>>>x.spam = 88>>>x.spam, y.spam, SharedData.spam(88, 99, 99)
class MixedNames:# Define classdata = 'spam'# Assign class attrdef __init__(self, value):# Assign method nameself.data = value# Assign instance attrdef display(self): print(self.data, MixedNames.data)# Instance attr, class attr
>>>x = MixedNames(1)# Make two instance objects>>>y = MixedNames(2)# Each has its own data>>>x.display(); y.display()# self.data differs, MixedNames.data is the same1 spam 2 spam
instance.method(args...)
class.method(instance,args...)
class NextClass:# Define classdef printer(self, text):# Define methodself.message = text# Change instanceprint(self.message)# Access instance
>>>x = NextClass()# Make instance>>>x.printer('instance call')# Call its methodinstance call >>>x.message# Instance changed'instance call'
>>>NextClass.printer(x, 'class call')# Direct class callclass call >>>x.message# Instance changed again'class call'
>>> NextClass.printer('bad call')
TypeError: unbound method printer() must be called with NextClass instance...class Super:
def __init__(self, x):
...default code...
class Sub(Super):
def __init__(self, x, y):
Super.__init__(self, x) # Run superclass __init__
...custom code... # Do my init actions
I = Sub(1, 2)self attributes in methods.class statements.class statement header.>>>class Super:def method(self):print('in Super.method')>>>class Sub(Super):def method(self):# Override methodprint('starting Sub.method')# Add actions hereSuper.method(self)# Run default actionprint('ending Sub.method')
>>>x = Super()# Make a Super instance>>>x.method()# Runs Super.methodin Super.method >>>x = Sub()# Make a Sub instance>>>x.method()# Runs Sub.method, calls Super.methodstarting Sub.method in Super.method ending Sub.method
SuperDefines amethodfunction and adelegatethat expects anactionin a subclass.
InheritorDoesn’t provide any new names, so it gets everything defined inSuper.
ReplacerOverridesSuper’smethodwith a version of its own.
ExtenderCustomizesSuper’smethodby overriding and calling back to run the default.
ProviderImplements theactionmethod expected bySuper’sdelegatemethod.
class Super:
def method(self):
print('in Super.method') # Default behavior
def delegate(self):
self.action() # Expected to be defined
class Inheritor(Super): # Inherit method verbatim
pass
class Replacer(Super): # Replace method completely
def method(self):
print('in Replacer.method')
class Extender(Super): # Extend method behavior
def method(self):
print('starting Extender.method')
Super.method(self)
print('ending Extender.method')
class Provider(Super): # Fill in a required method
def action(self):
print('in Provider.action')
if __name__ == '__main__':
for klass in (Inheritor, Replacer, Extender):
print('\n' + klass.__name__ + '...')
klass().method()
print('\nProvider...')
x = Provider()
x.delegate()% python specialize.py
Inheritor...
in Super.method
Replacer...
in Replacer.method
Extender...
starting Extender.method
in Super.method
ending Extender.method
Provider...
in Provider.actionx.delegate
call, Python finds the delegate
method in Super by searching the
Provider instance and above. The
instance x is passed into the
method’s self argument as
usual.Super.delegate
method, self.action invokes a
new, independent inheritance search of self and above. Because self references a Provider instance, the action method is located in the Provider subclass.class Super:
def delegate(self):
self.action()
def action(self):
assert False, 'action must be defined!' # If this version is called
>>> X = Super()
>>> X.delegate()
AssertionError: action must be defined!class Super:
def delegate(self):
self.action()
def action(self):
raise NotImplementedError('action must be defined!')
>>> X = Super()
>>> X.delegate()
NotImplementedError: action must be defined!>>>class Sub(Super): pass>>>X = Sub()>>>X.delegate()NotImplementedError: action must be defined! >>>class Sub(Super):def action(self): print('spam')>>>X = Sub()>>>X.delegate()spam
from abc import ABCMeta, abstractmethod
class Super(metaclass=ABCMeta):
@abstractmethod
def method(self, ...):
passclass Super:
__metaclass__ = ABCMeta
@abstractmethod
def method(self, ...):
pass>>>from abc import ABCMeta, abstractmethod>>> >>>class Super(metaclass=ABCMeta):def delegate(self):self.action()@abstractmethoddef action(self):pass>>>X = Super()TypeError: Can't instantiate abstract class Super with abstract methods action >>>class Sub(Super): pass>>>X = Sub()TypeError: Can't instantiate abstract class Sub with abstract methods action >>>class Sub(Super):def action(self): print('spam')>>>X = Sub()>>>X.delegate()spam
X)
deal with scopes.object.X) use object namespaces.X =
value)Makes names local by default: creates or changes the nameXin the current local scope, unless declared global (or nonlocal in 3.X).
X)Looks for the nameXin the current local scope, then any and all enclosing functions, then the current global scope, then the built-in scope, per the LEGB rule. Enclosing classes are not searched: class names are fetched as object attributes instead.
object.X =
value)Creates or alters the attribute nameXin the namespace of theobjectbeing qualified, and none other. Inheritance-tree climbing happens only on attribute reference, not on attribute assignment.
object.X)For class-based objects, searches for the attribute nameXinobject, then in all accessible classes above it, using the inheritance search procedure. For nonclass objects such as modules, fetchesXfromobjectdirectly.
# File manynames.pyX = 11# Global (module) name/attribute (X, or manynames.X)def f(): print(X)# Access global X (11)def g(): X = 22# Local (function) variable (X, hides module X)print(X) class C: X = 33# Class attribute (C.X)def m(self): X = 44# Local variable in method (X)self.X = 55# Instance attribute (instance.X)
# manynames.py, continuedif __name__ == '__main__': print(X)# 11: module (a.k.a. manynames.X outside file)f()# 11: globalg()# 22: localprint(X)# 11: module name unchangedobj = C()# Make instanceprint(obj.X)# 33: class name inherited by instanceobj.m()# Attach attribute name X to instance nowprint(obj.X)# 55: instanceprint(C.X)# 33: class (a.k.a. obj.X if no X in instance)#print(C.m.X)# FAILS: only visible in method#print(g.X)# FAILS: only visible in function
# otherfile.pyimport manynames X = 66 print(X)# 66: the global hereprint(manynames.X)# 11: globals become attributes after importsmanynames.f()# 11: manynames's X, not the one here!manynames.g()# 22: local in other file's functionprint(manynames.C.X)# 33: attribute of class in other moduleI = manynames.C() print(I.X)# 33: still from class hereI.m() print(I.X)# 55: now from instance!
X = 11# Global in moduledef g1(): print(X)# Reference global in module (11)def g2(): global X X = 22# Change global in moduledef h1(): X = 33# Local in functiondef nested(): print(X)# Reference local in enclosing scope (33)def h2(): X = 33# Local in functiondef nested(): nonlocal X# Python 3.X statementX = 44# Change local in enclosing scope
X = 1 def nester(): print(X)# Global: 1class C: print(X)# Global: 1def method1(self): print(X)# Global: 1def method2(self): X = 3# Hides globalprint(X)# Local: 3I = C() I.method1() I.method2() print(X)# Global: 1nester()# Rest: 1, 1, 1, 3print('-'*40)
X = 1 def nester(): X = 2# Hides globalprint(X)# Local: 2class C: print(X)# In enclosing def (nester): 2def method1(self): print(X)# In enclosing def (nester): 2def method2(self): X = 3# Hides enclosing (nester)print(X)# Local: 3I = C() I.method1() I.method2() print(X)# Global: 1nester()# Rest: 2, 2, 2, 3print('-'*40)
X = 1 def nester(): X = 2# Hides globalprint(X)# Local: 2class C: X = 3# Class local hides nester's: C.X or I.X (not scoped)print(X)# Local: 3def method1(self): print(X)# In enclosing def (not 3 in class!): 2print(self.X)# Inherited class local: 3def method2(self): X = 4# Hides enclosing (nester, not class)print(X)# Local: 4self.X = 5# Hides classprint(self.X)# Located in instance: 5I = C() I.method1() I.method2() print(X)# Global: 1nester()# Rest: 2, 3, 2, 3, 4, 5print('-'*40)
>>>class Super:def hello(self):self.data1 = 'spam'>>>class Sub(Super):def hola(self):self.data2 = 'eggs'
>>>X = Sub()>>>X.__dict__# Instance namespace dict{} >>>X.__class__# Class of instance<class '__main__.Sub'> >>>Sub.__bases__# Superclasses of class(<class '__main__.Super'>,) >>>Super.__bases__# () empty tuple in Python 2.X(<class 'object'>,)
>>>Y = Sub()>>>X.hello()>>>X.__dict__{'data1': 'spam'} >>>X.hola()>>>X.__dict__{'data2': 'eggs', 'data1': 'spam'} >>>list(Sub.__dict__.keys())['__qualname__', '__module__', '__doc__', 'hola'] >>>list(Super.__dict__.keys())['__module__', 'hello', '__dict__', '__qualname__', '__doc__', '__weakref__'] >>>Y.__dict__{}
>>>X.data1, X.__dict__['data1']('spam', 'spam') >>>X.data3 = 'toast'>>>X.__dict__{'data2': 'eggs', 'data3': 'toast', 'data1': 'spam'} >>>X.__dict__['data3'] = 'ham'>>>X.data3'ham'
#!python
"""
classtree.py: Climb inheritance trees using namespace links,
displaying higher superclasses with indentation for height
"""
def classtree(cls, indent):
print('.' * indent + cls.__name__) # Print class name here
for supercls in cls.__bases__: # Recur to all superclasses
classtree(supercls, indent+3) # May visit super > once
def instancetree(inst):
print('Tree of %s' % inst) # Show instance
classtree(inst.__class__, 3) # Climb to its class
def selftest():
class A: pass
class B(A): pass
class C(A): pass
class D(B,C): pass
class E: pass
class F(D,E): pass
instancetree(B())
instancetree(F())
if __name__ == '__main__': selftest()C:\code> c:\python27\python classtree.py
Tree of <__main__.B instance at 0x00000000022C3A88>
...B
......A
Tree of <__main__.F instance at 0x00000000022C3A88>
...F
......D
.........B
............A
.........C
............A
......EC:\code> c:\python33\python classtree.py
Tree of <__main__.selftest.<locals>.B object at 0x00000000029216A0>
...B
......A
.........object
Tree of <__main__.selftest.<locals>.F object at 0x00000000029216A0>
...F
......D
.........B
............A
...............object
.........C
............A
...............object
......E
.........objectC:\code>c:\python33\python>>>class Emp: pass>>>class Person(Emp): pass>>>bob = Person()>>>import classtree>>>classtree.instancetree(bob)Tree of <__main__.Person object at 0x000000000298B6D8> ...Person ......Emp .........object
"I am: docstr.__doc__"
def func(args):
"I am: docstr.func.__doc__"
pass
class spam:
"I am: spam.__doc__ or docstr.spam.__doc__ or self.__doc__"
def method(self):
"I am: spam.method.__doc__ or self.method.__doc__"
print(self.__doc__)
print(self.method.__doc__)>>>import docstr>>>docstr.__doc__'I am: docstr.__doc__' >>>docstr.func.__doc__'I am: docstr.func.__doc__' >>>docstr.spam.__doc__'I am: spam.__doc__ or docstr.spam.__doc__ or self.__doc__' >>>docstr.spam.method.__doc__'I am: spam.method.__doc__ or self.method.__doc__' >>>x = docstr.spam()>>>x.method()I am: spam.__doc__ or docstr.spam.__doc__ or self.__doc__ I am: spam.method.__doc__ or self.method.__doc__
>>> help(docstr)
Help on module docstr:
NAME
docstr - I am: docstr.__doc__
FILE
c:\code\docstr.py
CLASSES
spam
class spam
| I am: spam.__doc__ or docstr.spam.__doc__ or self.__doc__
|
| Methods defined here:
|
| method(self)
| I am: spam.method.__doc__ or self.method.__doc__
FUNCTIONS
func(args)
I am: docstr.func.__doc__class
statementsclass
statement?__init__ method in a superclass?X =
Y) appears at the top level of a class statement, it attaches a data
attribute to the class (Class.X). Like all class attributes, this will be
shared by all instances; data attributes are not callable method
functions, though.__init__ method in a superclass if it
defines an __init__ constructor of
its own and still wants the superclass’s construction code to run.
Python itself automatically runs just one
constructor — the lowest one in the tree. Superclass constructors are
usually called through the class name, passing in the self instance manually:
Superclass.__init__(self, ...).self
instance to the superclass’s version of the method manually:
Superclass.method(self, ...).class
statement is run.1 If you’ve used C++ you may recognize this as similar to the
notion of C++’s “static” data members — members that are stored in the
class, independent of instances. In Python, it’s nothing special:
all class attributes are just names assigned in the class statement, whether they happen to
reference functions (C++’s
“methods”) or something else (C++’s “members”). In Chapter 32, we’ll also meet Python static
methods (akin to those in C++), which are just self-less functions that usually process
class attributes.
2 Unless the class has redefined the attribute assignment
operation to do something unique with the __setattr__ operator overloading method
(discussed in Chapter 30), or uses
advanced attribute tools such as properties and
descriptors (discussed in Chapter 32 and Chapter 38). Much of this chapter presents the
normal case, which suffices at this point in the book, but as we’ll
see later, Python hooks allow programs to deviate from the norm
often.
3 On a related note, you can also code multiple __init__ methods within the same class,
but only the last definition will be used; see Chapter 31 for more details on multiple
method definitions.
4 Two fine points here: first, this description isn’t 100%
complete, because we can also create instance and class attributes
by assigning them to objects outside class statements — but that’s a much less
common and sometimes more error-prone approach (changes aren’t
isolated to class statements). In
Python, all attributes are always accessible by default. We’ll talk
more about attribute name privacy in Chapter 30 when we study __setattr__, in Chapter 31 when we meet __X names, and
again in Chapter 39, where we’ll implement it
with a class decorator.
Second, as also noted in Chapter 27, the full
inheritance story grows more convoluted when
advanced topics such as metaclasses and
descriptors are added to the mix — and we’re
deferring a formal definition until Chapter 40
for this reason. In common usage, though, it’s simply a way to
redefine, and hence customize, behavior coded in classes.
# File number.pyclass Number: def __init__(self, start):# On Number(start)self.data = start def __sub__(self, other):# On instance - otherreturn Number(self.data - other)# Result is a new instance>>>from number import Number# Fetch class from module>>>X = Number(5)# Number.__init__(X, 5)>>>Y = X - 2# Number.__sub__(X, 2)>>>Y.data# Y is new Number instance3
| Method | Implements | Called for |
|---|---|---|
c:\code>py −3 -m timeit -n 1000 -r 5-s "L = list(range(100))" "x = L.__len__()"1000 loops, best of 5: 0.134 usec per loop c:\code>py −3 -m timeit -n 1000 -r 5-s "L = list(range(100))" "x = len(L)"1000 loops, best of 5: 0.063 usec per loop c:\code>py −2 -m timeit -n 1000 -r 5-s "L = list(range(100))" "x = L.__len__()"1000 loops, best of 5: 0.117 usec per loop c:\code>py −2 -m timeit -n 1000 -r 5-s "L = list(range(100))" "x = len(L)"1000 loops, best of 5: 0.0596 usec per loop
>>>class Indexer:def __getitem__(self, index):return index ** 2>>>X = Indexer()>>>X[2]# X[i] calls X.__getitem__(i)4 >>>for i in range(5):print(X[i], end=' ')# Runs __getitem__(X, i) each time0 1 4 9 16
>>>L = [5, 6, 7, 8, 9]>>>L[2:4]# Slice with slice syntax: 2..(4-1)[7, 8] >>>L[1:][6, 7, 8, 9] >>>L[:-1][5, 6, 7, 8] >>>L[::2][5, 7, 9]
>>>L[slice(2, 4)]# Slice with slice objects[7, 8] >>>L[slice(1, None)][6, 7, 8, 9] >>>L[slice(None, −1)][5, 6, 7, 8] >>>L[slice(None, None, 2)][5, 7, 9]
>>>class Indexer:data = [5, 6, 7, 8, 9]def __getitem__(self, index):# Called for index or sliceprint('getitem:', index)return self.data[index]# Perform index or slice>>>X = Indexer()>>>X[0]# Indexing sends __getitem__ an integergetitem: 0 5 >>>X[1]getitem: 1 6 >>>X[-1]getitem: −1 9
>>>X[2:4]# Slicing sends __getitem__ a slice objectgetitem: slice(2, 4, None) [7, 8] >>>X[1:]getitem: slice(1, None, None) [6, 7, 8, 9] >>>X[:-1]getitem: slice(None, −1, None) [5, 6, 7, 8] >>>X[::2]getitem: slice(None, None, 2) [5, 7, 9]
>>>class Indexer:def __getitem__(self, index):if isinstance(index, int):# Test usage modeprint('indexing', index)else:print('slicing', index.start, index.stop, index.step)>>>X = Indexer()>>>X[99]indexing 99 >>>X[1:99:2]slicing 1 99 2 >>>X[1:]slicing 1 None None
class IndexSetter:
def __setitem__(self, index, value): # Intercept index or slice assignment
...
self.data[index] = value # Assign index or sliceC:\code>c:\python27\python>>>class Slicer:def __getitem__(self, index): print indexdef __getslice__(self, i, j): print i, jdef __setslice__(self, i, j,seq): print i, j,seq>>>Slicer()[1]# Runs __getitem__ with int, like 3.X1 >>>Slicer()[1:9]# Runs __getslice__ if present, else __getitem__1 9 >>>Slicer()[1:9:2]# Runs __getitem__ with slice(), like 3.X!slice(1, 9, 2)
>>>class C:def __index__(self):return 255>>>X = C()>>>hex(X)# Integer value'0xff' >>>bin(X)'0b11111111' >>>oct(X)'0o377'
>>>('C' * 256)[255]'C' >>>('C' * 256)[X]# As index (not X[i])'C' >>>('C' * 256)[X:]# As index (not X[i:])'C'
>>>class StepperIndex:def __getitem__(self, i):return self.data[i]>>>X = StepperIndex()# X is a StepperIndex object>>>X.data = "Spam">>> >>>X[1]# Indexing calls __getitem__'p' >>>for item in X:# for loops call __getitem__print(item, end=' ')# for indexes items 0..NS p a m
>>>'p' in X# All call __getitem__ tooTrue >>>[c for c in X]# List comprehension['S', 'p', 'a', 'm'] >>>list(map(str.upper, X))# map calls (use list() in 3.X)['S', 'P', 'A', 'M'] >>>(a, b, c, d) = X# Sequence assignments>>>a, c, d('S', 'a', 'm') >>>list(X), tuple(X), ''.join(X)# And so on...(['S', 'p', 'a', 'm'], ('S', 'p', 'a', 'm'), 'Spam') >>>X<__main__.StepperIndex object at 0x000000000297B630>
# File squares.pyclass Squares: def __init__(self, start, stop):# Save state when createdself.value = start - 1 self.stop = stop def __iter__(self):# Get iterator object on iterreturn self def __next__(self):# Return a square on each iterationif self.value == self.stop:# Also called by next built-inraise StopIteration self.value += 1 return self.value ** 2
%python>>>from squares import Squares>>>for i in Squares(1, 5):# for calls iter, which calls __iter__print(i, end=' ')# Each iteration calls __next__1 4 9 16 25
>>>X = Squares(1, 5)# Iterate manually: what loops do>>>I = iter(X)# iter calls __iter__>>>next(I)# next calls __next__ (in 3.X)1 >>>next(I)4...more omitted...>>>next(I)25 >>>next(I)# Can catch this in try statementStopIteration
>>>X = Squares(1, 5)>>>X[1]TypeError: 'Squares' object does not support indexing >>>list(X)[1]4
>>>X = Squares(1, 5)# Make an iterable with state>>>[n for n in X]# Exhausts items: __iter__ returns self[1, 4, 9, 16, 25] >>>[n for n in X]# Now it's empty: __iter__ returns same self[] >>>[n for n in Squares(1, 5)]# Make a new iterable object[1, 4, 9, 16, 25] >>>list(Squares(1, 3))# A new object for each new __iter__ call[1, 4, 9]
>>>36 in Squares(1, 10)# Other iteration contextsTrue >>>a, b, c = Squares(1, 3)# Each calls __iter__ and then __next__>>>a, b, c(1, 4, 9) >>>':'.join(map(str, Squares(1, 5)))'1:4:9:16:25'
>>>X = Squares(1, 5)>>>tuple(X), tuple(X)# Iterator exhausted in second tuple()((1, 4, 9, 16, 25), ()) >>>X = list(Squares(1, 5))>>>tuple(X), tuple(X)((1, 4, 9, 16, 25), (1, 4, 9, 16, 25))
>>>def gsquares(start, stop):for i in range(start, stop + 1):yield i ** 2>>>for i in gsquares(1, 5):print(i, end=' ')1 4 9 16 25 >>>for i in (x ** 2 for x in range(1, 6)):print(i, end=' ')1 4 9 16 25
>>> [x ** 2 for x in range(1, 6)]
[1, 4, 9, 16, 25]>>>S = 'ace'>>>for x in S:for y in S:print(x + y, end=' ')aa ac ae ca cc ce ea ec ee
#!python3# File skipper.pyclass SkipObject: def __init__(self, wrapped):# Save item to be usedself.wrapped = wrapped def __iter__(self): return SkipIterator(self.wrapped)# New iterator each timeclass SkipIterator: def __init__(self, wrapped): self.wrapped = wrapped# Iterator state informationself.offset = 0 def __next__(self): if self.offset >= len(self.wrapped):# Terminate iterationsraise StopIteration else: item = self.wrapped[self.offset]# else return and skipself.offset += 2 return item if __name__ == '__main__': alpha = 'abcdef' skipper = SkipObject(alpha)# Make container objectI = iter(skipper)# Make an iterator on itprint(next(I), next(I), next(I))# Visit offsets 0, 2, 4for x in skipper:# for calls __iter__ automaticallyfor y in skipper:# Nested fors call __iter__ again each timeprint(x + y, end=' ')# Each iterator has its own state, offset
#!python
from __future__ import print_function # 2.X/3.X compatibility
...
class SkipIterator:
...
def __next__(self):
...
next = __next__ # 2.X/3.X compatibility% python skipper.py
a c e
aa ac ae ca cc ce ea ec ee>>>S = 'abcdef'>>>for x in S[::2]:for y in S[::2]:# New objects on each iterationprint(x + y, end=' ')aa ac ae ca cc ce ea ec ee
>>>S = 'abcdef'>>>S = S[::2]>>>S'ace' >>>for x in S:for y in S:# Same object, new iteratorsprint(x + y, end=' ')aa ac ae ca cc ce ea ec ee
>>>def gen(x):for i in range(x): yield i ** 2>>>G = gen(5)# Create a generator with __iter__ and __next__>>>G.__iter__() == G# Both methods exist on the same objectTrue >>>I = iter(G)# Runs __iter__: generator returns itself>>>next(I), next(I)# Runs __next__ (next in 2.X)(0, 1) >>>list(gen(5))# Iteration contexts automatically run iter and next[0, 1, 4, 9, 16]
# File squares_yield.pyclass Squares:# __iter__ + yield generatordef __init__(self, start, stop):# __next__ is automatic/impliedself.start = start self.stop = stop def __iter__(self): for value in range(self.start, self.stop + 1): yield value ** 2
%python>>>from squares_yield import Squares>>>for i in Squares(1, 5): print(i, end=' ')1 4 9 16 25
>>>S = Squares(1, 5)# Runs __init__: class saves instance state>>>S<squares_yield.Squares object at 0x000000000294B630> >>>I = iter(S)# Runs __iter__: returns a generator>>>I<generator object __iter__ at 0x00000000029A8CF0> >>>next(I)1 >>>next(I)# Runs generator's __next__4...etc...>>>next(I)# Generator has both instance and local scope stateStopIteration
class Squares:# Non __iter__ equivalent (squares_manual.py)def __init__(...): ... def gen(self): for value in range(self.start, self.stop + 1): yield value ** 2 %python>>>from squares_manual import Squares>>>for i in Squares(1, 5).gen(): print(i, end=' ')...same results... >>>S = Squares(1, 5)>>>I = iter(S.gen())# Call generator manually for iterable/iterator>>>next(I)...same results...
__iter__, iteration
triggers __iter__, which returns
a new generator with __next__.__iter__, your code
calls to make a generator, which returns itself for __iter__.%python>>>from squares_yield import Squares# Using the __iter__/yield Squares>>>S = Squares(1, 5)>>>I = iter(S)>>>next(I); next(I)1 4 >>>J = iter(S)# With yield, multiple iterators automatic>>>next(J)1 >>>next(I)# I is independent of J: own local state9
>>>S = Squares(1, 3)>>>for i in S:# Each for calls __iter__for j in S:print('%s:%s' % (i, j), end=' ')1:1 1:4 1:9 4:1 4:4 4:9 9:1 9:4 9:9
# File squares_nonyield.pyclass Squares: def __init__(self, start, stop):# Non-yield generatorself.start = start# Multiscans: extra objectself.stop = stop def __iter__(self): return SquaresIter(self.start, self.stop) class SquaresIter: def __init__(self, start, stop): self.value = start - 1 self.stop = stop def __next__(self): if self.value == self.stop: raise StopIteration self.value += 1 return self.value ** 2
%python>>>from squares_nonyield import Squares>>>for i in Squares(1, 5): print(i, end=' ')1 4 9 16 25 >>> >>>S = Squares(1, 5)>>>I = iter(S)>>>next(I); next(I)1 4 >>>J = iter(S)# Multiple iterators without yield>>>next(J)1 >>>next(I)9 >>>S = Squares(1, 3)>>>for i in S:# Each for calls __iter__for j in S:print('%s:%s' % (i, j), end=' ')1:1 1:4 1:9 4:1 4:4 4:9 9:1 9:4 9:9
# File skipper_yield.pyclass SkipObject:# Another __iter__ + yield generatordef __init__(self, wrapped):# Instance scope retained normallyself.wrapped = wrapped# Local scope state saved autodef __iter__(self): offset = 0 while offset < len(self.wrapped): item = self.wrapped[offset] offset += 2 yield item
%python>>>from skipper_yield import SkipObject>>>skipper = SkipObject('abcdef')>>>I = iter(skipper)>>>next(I); next(I); next(I)'a' 'c' 'e' >>>for x in skipper:# Each for calls __iter__: new auto generatorfor y in skipper:print(x + y, end=' ')aa ac ae ca cc ce ea ec ee
__lt__ for “less than” if present, or else
the general __cmp__. Python 3.X
uses only specific methods, not __cmp__, as discussed later in this
chapter.__bool__ first (to give an explicit True/False result), and if it’s absent fall back
on the more general __len__ (a
nonzero length means True). As
we’ll also see later in this chapter, Python 2.X works the same but
uses the name __nonzero__ instead
of __bool__.# File contains.pyfrom __future__ import print_function# 2.X/3.X compatibilityclass Iters: def __init__(self, value): self.data = value def __getitem__(self, i):# Fallback for iterationprint('get[%s]:' % i, end='')# Also for index, slicereturn self.data[i] def __iter__(self):# Preferred for iterationprint('iter=> ', end='')# Allows only one active iteratorself.ix = 0 return self def __next__(self): print('next:', end='') if self.ix == len(self.data): raise StopIteration item = self.data[self.ix] self.ix += 1 return item def __contains__(self, x):# Preferred for 'in'print('contains: ', end='') return x in self.data next = __next__# 2.X/3.X compatibilityif __name__ == '__main__': X = Iters([1, 2, 3, 4, 5])# Make instanceprint(3 in X)# Membershipfor i in X:# for loopsprint(i, end=' | ') print() print([i ** 2 for i in X])# Other iteration contextsprint( list(map(bin, X)) ) I = iter(X)# Manual iteration (what other contexts do)while True: try: print(next(I), end=' @ ') except StopIteration: break
class Iters:
def __init__(self, value):
self.data = value
def __getitem__(self, i): # Fallback for iteration
print('get[%s]:' % i, end='') # Also for index, slice
return self.data[i]
def __iter__(self): # Preferred for iteration
print('iter=> next:', end='') # Allows multiple active iterators
for x in self.data: # no __next__ to alias to next
yield x
print('next:', end='')
def __contains__(self, x): # Preferred for 'in'
print('contains: ', end='')
return x in self.datacontains: True iter=> next:1 | next:2 | next:3 | next:4 | next:5 | next: iter=> next:next:next:next:next:next:[1, 4, 9, 16, 25] iter=> next:next:next:next:next:next:['0b1', '0b10', '0b11', '0b100', '0b101'] iter=> next:1 @ next:2 @ next:3 @ next:4 @ next:5 @ next:
iter=> next:next:next:True iter=> next:1 | next:2 | next:3 | next:4 | next:5 | next: iter=> next:next:next:next:next:next:[1, 4, 9, 16, 25] iter=> next:next:next:next:next:next:['0b1', '0b10', '0b11', '0b100', '0b101'] iter=> next:1 @ next:2 @ next:3 @ next:4 @ next:5 @ next:
get[0]:get[1]:get[2]:True get[0]:1 | get[1]:2 | get[2]:3 | get[3]:4 | get[4]:5 | get[5]: get[0]:get[1]:get[2]:get[3]:get[4]:get[5]:[1, 4, 9, 16, 25] get[0]:get[1]:get[2]:get[3]:get[4]:get[5]:['0b1', '0b10', '0b11', '0b100','0b101'] get[0]:1 @ get[1]:2 @ get[2]:3 @ get[3]:4 @ get[4]:5 @ get[5]:
>>>from contains import Iters>>>X = Iters('spam')# Indexing>>>X[0]# __getitem__(0)get[0]:'s' >>>'spam'[1:]# Slice syntax'pam' >>>'spam'[slice(1, None)]# Slice object'pam' >>>X[1:]# __getitem__(slice(..))get[slice(1, None, None)]:'pam' >>>X[:-1]get[slice(None, −1, None)]:'spa' >>>list(X)# And iteration too!iter=> next:next:next:next:next:['s', 'p', 'a', 'm']
>>>class Empty:def __getattr__(self, attrname):# On self.undefinedif attrname == 'age':return 40else:raise AttributeError(attrname)>>>X = Empty()>>>X.age40 >>>X.name...error text omitted...AttributeError: name
>>>class Accesscontrol:def __setattr__(self, attr, value):if attr == 'age':self.__dict__[attr] = value + 10# Not self.name=val or setattrelse:raise AttributeError(attr + ' not allowed')>>>X = Accesscontrol()>>>X.age = 40# Calls __setattr__>>>X.age50 >>>X.name = 'Bob'...text omitted...AttributeError: name not allowed
self.age = value + 10# Loopssetattr(self, attr, value + 10)# Loops (attr is 'age')
self.other = 99 # Recurs but doesn't loop: failsself.__dict__[attr] = value + 10# OK: doesn't loopobject.__setattr__(self, attr, value + 10)# OK: doesn't loop (new-style only)
__getattribute__ method
intercepts all attribute fetches,
not just those that are undefined, but when using it you must be
more cautious than with __getattr__ to avoid loops.property built-in
function allows us to associate methods with fetch and set
operations on a specific class
attribute.__get__
and __set__ methods of a class with accesses to a
specific class attribute.class PrivateExc(Exception): pass# More on exceptions in Part VIIclass Privacy: def __setattr__(self, attrname, value):# On self.attrname = valueif attrname in self.privates: raise PrivateExc(attrname, self)# Make, raise user-define exceptelse: self.__dict__[attrname] = value# Avoid loops by using dict keyclass Test1(Privacy): privates = ['age'] class Test2(Privacy): privates = ['name', 'pay'] def __init__(self): self.__dict__['name'] = 'Tom'# To do better, see Chapter 39!if __name__ == '__main__': x = Test1() y = Test2() x.name = 'Bob' # Works #y.name = 'Sue' # Fails print(x.name) y.age = 30 # Works #x.age = 40 # Fails print(y.age)
>>>class adder:def __init__(self, value=0):self.data = value# Initialize datadef __add__(self, other):self.data += other# Add other in place (bad form?)>>>x = adder()# Default displays>>>print(x)<__main__.adder object at 0x00000000029736D8> >>>x<__main__.adder object at 0x00000000029736D8>
>>>class addrepr(adder):# Inherit __init__, __add__def __repr__(self):# Add string representationreturn 'addrepr(%s)' % self.data# Convert to as-code string>>>x = addrepr(2)# Runs __init__>>>x + 1# Runs __add__ (x.add() better?)>>>x# Runs __repr__addrepr(3) >>>print(x)# Runs __repr__addrepr(3) >>>str(x), repr(x)# Runs __repr__ for both('addrepr(3)', 'addrepr(3)')
__str__ is tried first for
the print operation and the
str built-in function (the
internal equivalent of which print runs). It generally should return a
user-friendly display.__repr__ is used in all
other contexts: for interactive echoes, the repr function, and nested appearances, as
well as by print and str if no __str__ is present. It should generally
return an as-code string that could be used to re-create the object,
or a detailed display for developers.>>>class addstr(adder):def __str__(self):# __str__ but no __repr__return '[Value: %s]' % self.data# Convert to nice string>>>x = addstr(3)>>>x + 1>>>x# Default __repr__<__main__.addstr object at 0x00000000029738D0> >>>print(x)# Runs __str__[Value: 4] >>>str(x), repr(x)('[Value: 4]', '<__main__.addstr object at 0x00000000029738D0>')
>>>class addboth(adder):def __str__(self):return '[Value: %s]' % self.data# User-friendly stringdef __repr__(self):return 'addboth(%s)' % self.data# As-code string>>>x = addboth(4)>>>x + 1>>>x# Runs __repr__addboth(5) >>>print(x)# Runs __str__[Value: 5] >>>str(x), repr(x)('[Value: 5]', 'addboth(5)')
>>>class Printer:def __init__(self, val):self.val = valdef __str__(self):# Used for instance itselfreturn str(self.val)# Convert to a string result>>>objs = [Printer(2), Printer(3)]>>>for x in objs: print(x)# __str__ run when instance printed# But not when instance is in a list!2 3 >>>print(objs)[<__main__.Printer object at 0x000000000297AB38>, <__main__.Printer obj...etc...>] >>>objs[<__main__.Printer object at 0x000000000297AB38>, <__main__.Printer obj...etc...>]
>>>class Printer:def __init__(self, val):self.val = valdef __repr__(self):# __repr__ used by print if no __str__return str(self.val)# __repr__ used if echoed or nested>>>objs = [Printer(2), Printer(3)]>>>for x in objs: print(x)# No __str__: runs __repr__2 3 >>>print(objs)# Runs __repr__, not __str__[2, 3] >>>objs[2, 3]
>>>class Adder:def __init__(self, value=0):self.data = valuedef __add__(self, other):return self.data + other>>>x = Adder(5)>>>x + 27 >>>2 + xTypeError: unsupported operand type(s) for +: 'int' and 'Adder'
class Commuter1:
def __init__(self, val):
self.val = val
def __add__(self, other):
print('add', self.val, other)
return self.val + other
def __radd__(self, other):
print('radd', self.val, other)
return other + self.val
>>> from commuter import Commuter1
>>> x = Commuter1(88)
>>> y = Commuter1(99)
>>> x + 1 # __add__: instance + noninstance
add 88 1
89
>>> 1 + y # __radd__: noninstance + instance
radd 99 1
100
>>> x + y # __add__: instance + instance, triggers __radd__
add 88 <commuter.Commuter1 object at 0x00000000029B39E8>
radd 99 88
187class Commuter2:
def __init__(self, val):
self.val = val
def __add__(self, other):
print('add', self.val, other)
return self.val + other
def __radd__(self, other):
return self.__add__(other) # Call __add__ explicitly
class Commuter3:
def __init__(self, val):
self.val = val
def __add__(self, other):
print('add', self.val, other)
return self.val + other
def __radd__(self, other):
return self + other # Swap order and re-add
class Commuter4:
def __init__(self, val):
self.val = val
def __add__(self, other):
print('add', self.val, other)
return self.val + other
__radd__ = __add__ # Alias: cut out the middlemanclass Commuter5:# Propagate class type in resultsdef __init__(self, val): self.val = val def __add__(self, other): if isinstance(other, Commuter5):# Type test to avoid object nestingother = other.val return Commuter5(self.val + other)# Else + result is another Commuterdef __radd__(self, other): return Commuter5(other + self.val) def __str__(self): return '<Commuter5: %s>' % self.val >>>from commuter import Commuter5>>>x = Commuter5(88)>>>y = Commuter5(99)>>>print(x + 10)# Result is another Commuter instance<Commuter5: 98> >>>print(10 + y)<Commuter5: 109> >>>z = x + y# Not nested: doesn't recur to __radd__>>>print(z)<Commuter5: 187> >>>print(z + 10)<Commuter5: 197> >>>print(z + z)<Commuter5: 374> >>>print(z + z + 1)<Commuter5: 375>
>>>z = x + y# With isinstance test commented-out>>>print(z)<Commuter5: <Commuter5: 187>> >>>print(z + 10)<Commuter5: <Commuter5: 197>> >>>print(z + z)<Commuter5: <Commuter5: <Commuter5: <Commuter5: 374>>>> >>>print(z + z + 1)<Commuter5: <Commuter5: <Commuter5: <Commuter5: 375>>>>
#!python from __future__ import print_function# 2.X/3.X compatibility...classes defined here...if __name__ == '__main__': for klass in (Commuter1, Commuter2, Commuter3, Commuter4, Commuter5): print('-' * 60) x = klass(88) y = klass(99) print(x + 1) print(1 + y) print(x + y) c:\code>commuter.py------------------------------------------------------------ add 88 1 89 radd 99 1 100 add 88 <__main__.Commuter1 object at 0x000000000297F2B0> radd 99 88 187 ------------------------------------------------------------...etc...
>>>class Number:def __init__(self, val):self.val = valdef __iadd__(self, other):# __iadd__ explicit: x += yself.val += other# Usually returns selfreturn self>>>x = Number(5)>>>x += 1>>>x += 1>>>x.val7
>>>y = Number([1])# In-place change faster than +>>>y += [2]>>>y += [3]>>>y.val[1, 2, 3]
>>>class Number:def __init__(self, val):self.val = valdef __add__(self, other):# __add__ fallback: x = (x + y)return Number(self.val + other)# Propagates class type>>>x = Number(5)>>>x += 1>>>x += 1# And += does concatenation here>>>x.val7
>>>class Callee:def __call__(self, *pargs, **kargs):# Intercept instance callsprint('Called:', pargs, kargs)# Accept arbitrary arguments>>>C = Callee()>>>C(1, 2, 3)# C is a callable objectCalled: (1, 2, 3) {} >>>C(1, 2, 3, x=4, y=5)Called: (1, 2, 3) {'y': 5, 'x': 4}
class C:
def __call__(self, a, b, c=5, d=6): ... # Normals and defaults
class C:
def __call__(self, *pargs, **kargs): ... # Collect arbitrary arguments
class C:
def __call__(self, *pargs, d=6, **kargs): ... # 3.X keyword-only argumentX = C() X(1, 2)# Omit defaultsX(1, 2, 3, 4)# PositionalsX(a=1, b=2, d=4)# KeywordsX(*[1, 2], **dict(c=3, d=4))# Unpack arbitrary argumentsX(1, *(2,), c=3, **dict(d=4))# Mixed modes
>>>class Prod:def __init__(self, value):# Accept just one argumentself.value = valuedef __call__(self, other):return self.value * other>>>x = Prod(2)# "Remembers" 2 in state>>>x(3)# 3 (passed) * 2 (state)6 >>>x(4)8
>>>class Prod:def __init__(self, value):self.value = valuedef comp(self, other):return self.value * other>>>x = Prod(3)>>>x.comp(3)9 >>>x.comp(4)12
class Callback:
def __init__(self, color): # Function + state information
self.color = color
def __call__(self): # Support calls with no arguments
print('turn', self.color)# Handlerscb1 = Callback('blue')# Remember bluecb2 = Callback('green')# Remember greenB1 = Button(command=cb1)# Register handlersB2 = Button(command=cb2)
# Eventscb1()# Prints 'turn blue'cb2()# Prints 'turn green'
def callback(color):# Enclosing scope versus attrsdef oncall(): print('turn', color) return oncall cb3 = callback('yellow')# Handler to be registeredcb3()# On event: prints 'turn yellow'
cb4 = (lambda color='red': 'turn ' + color) # Defaults retain state too
print(cb4())class Callback:
def __init__(self, color): # Class with state information
self.color = color
def changeColor(self): # A normal named method
print('turn', self.color)
cb1 = Callback('blue')
cb2 = Callback('yellow')
B1 = Button(command=cb1.changeColor) # Bound method: reference, don't call
B2 = Button(command=cb2.changeColor) # Remembers function + self paircb1 = Callback('blue')
obj = cb1.changeColor # Registered event handler
obj() # On event prints 'turn blue'__add__/__radd__ pairings discussed earlier, there
are no right-side variants of comparison methods. Instead, reflective
methods are used when only one operand supports comparison (e.g., __lt__ and __gt__ are each other’s reflection).== does not
imply that != is false, for
example, so both __eq__ and
__ne__ should be defined to ensure
that both operators behave correctly.__cmp__
method is used by all comparisons if no more specific
comparison methods are defined; it returns a number that is less than,
equal to, or greater than zero, to signal less than, equal, and
greater than results for the comparison of its two arguments (self and another operand). This method often
uses the cmp(x, y) built-in to
compute its result. Both the __cmp__ method and the cmp built-in function are removed in Python
3.X: use the more specific methods instead.class C:
data = 'spam'
def __gt__(self, other): # 3.X and 2.X version
return self.data > other
def __lt__(self, other):
return self.data < other
X = C()
print(X > 'ham') # True (runs __gt__)
print(X < 'ham') # False (runs __lt__)class C:
data = 'spam' # 2.X only
def __cmp__(self, other): # __cmp__ not used in 3.X
return cmp(self.data, other) # cmp not defined in 3.X
X = C()
print(X > 'ham') # True (runs __cmp__)
print(X < 'ham') # False (runs __cmp__)class C:
data = 'spam'
def __cmp__(self, other):
return (self.data > other) - (self.data < other)>>>class Truth:def __bool__(self): return True>>>X = Truth()>>>if X: print('yes!')yes! >>>class Truth:def __bool__(self): return False>>>X = Truth()>>>bool(X)False
>>>class Truth:def __len__(self): return 0>>>X = Truth()>>>if not X: print('no!')no!
>>>class Truth:def __bool__(self): return True# 3.X tries __bool__ firstdef __len__(self): return 0# 2.X tries __len__ first>>>X = Truth()>>>if X: print('yes!')yes!
>>>class Truth:pass>>>X = Truth()>>>bool(X)True
C:\code>c:\python33\python>>>class C:def __bool__(self):print('in bool')return False>>>X = C()>>>bool(X)in bool False >>>if X: print(99)in bool
C:\code>c:\python27\python>>>class C:def __bool__(self):print('in bool')return False>>>X = C()>>>bool(X)True >>>if X: print(99)99
C:\code>c:\python27\python>>>class C:def __nonzero__(self):print('in nonzero')return False# Returns int (or True/False, same as 1/0)>>>X = C()>>>bool(X)in nonzero False >>>if X: print(99)in nonzero
>>>class Life:def __init__(self, name='unknown'):print('Hello ' + name)self.name = namedef live(self):print(self.name)def __del__(self):print('Goodbye ' + self.name)>>>brian = Life('Brian')Hello Brian >>>brian.live()Brian >>>brian = 'loretta'Goodbye Brian
__del__ can be tricky to use for even more
subtle reasons. Exceptions raised within it, for example, simply
print a warning message to sys.stderr (the
standard error stream) rather than triggering an exception event, because of
the unpredictable context under which it is run by the garbage
collector — it’s not always possible to know where such an exception
should be delivered.__del__ methods. Since this is relatively
obscure, we’ll ignore further details here; see Python’s standard
manuals’ coverage of both __del__
and the gc garbage collector
module for more information.__enter__ and __exit__ in with statement context managers.__get__ and __set__ class descriptor fetch/set
methods.__new__ object creation method in the
context of metaclasses.__getitem__ or __iter__. In all iteration contexts, Python
tries to use __iter__ first, which
returns an object that supports the iteration protocol with a __next__ method: if no __iter__ is found by inheritance search,
Python falls back on the __getitem__ indexing method, which is called
repeatedly, with successively higher indexes. If used, the yield statement can create the __next__ method automatically.__str__ and __repr__ methods implement object print
displays. The former is called by the print and str built-in functions; the latter is called
by print and str if there is no __str__, and always by the repr built-in, interactive echoes, and
nested appearances. That is, __repr__ is used everywhere, except by
print and str when a __str__ is defined. A __str__ is usually used for user-friendly
displays; __repr__ gives extra
details or the object’s as-code form.__getitem__ indexing method: it is called
with a slice object, instead of a simple integer index, and slice
objects may be passed on or inspected as needed. In Python 2.X,
__getslice__ (defunct in 3.X) may
be used for two-limit slices as well.__iadd__ first, and __add__ with an assignment second. The same
pattern holds true for all binary operators. The __radd__ method is also available for
right-side addition.class C:
def meth(self, x):
...
def meth(self, x, y, z):
...class C:
def meth(self, *args):
if len(args) == 1: # Branch on number arguments
...
elif type(arg[0]) == int: # Branch on argument types (or isinstance())
...class C:
def meth(self, x):
x.operation() # Assume x does the right thing# File employees.py (2.X + 3.X)from __future__ import print_function class Employee: def __init__(self, name, salary=0): self.name = name self.salary = salary def giveRaise(self, percent): self.salary = self.salary + (self.salary * percent) def work(self): print(self.name, "does stuff") def __repr__(self): return "<Employee: name=%s, salary=%s>" % (self.name, self.salary) class Chef(Employee): def __init__(self, name): Employee.__init__(self, name, 50000) def work(self): print(self.name, "makes food") class Server(Employee): def __init__(self, name): Employee.__init__(self, name, 40000) def work(self): print(self.name, "interfaces with customer") class PizzaRobot(Chef): def __init__(self, name): Chef.__init__(self, name) def work(self): print(self.name, "makes pizza") if __name__ == "__main__": bob = PizzaRobot('bob')# Make a robot named bobprint(bob)# Run inherited __repr__bob.work()# Run type-specific actionbob.giveRaise(0.20)# Give bob a 20% raiseprint(bob); print() for klass in Employee, Chef, Server, PizzaRobot: obj = klass(klass.__name__) obj.work()
c:\code> python employees.py
<Employee: name=bob, salary=50000>
bob makes pizza
<Employee: name=bob, salary=60000.0>
Employee does stuff
Chef makes food
Server interfaces with customer
PizzaRobot makes pizza# File pizzashop.py (2.X + 3.X)from __future__ import print_function from employees import PizzaRobot, Server class Customer: def __init__(self, name): self.name = name def order(self, server): print(self.name, "orders from", server) def pay(self, server): print(self.name, "pays for item to", server) class Oven: def bake(self): print("oven bakes") class PizzaShop: def __init__(self): self.server = Server('Pat')# Embed other objectsself.chef = PizzaRobot('Bob')# A robot named bobself.oven = Oven() def order(self, name): customer = Customer(name)# Activate other objectscustomer.order(self.server)# Customer orders from serverself.chef.work() self.oven.bake() customer.pay(self.server) if __name__ == "__main__": scene = PizzaShop()# Make the compositescene.order('Homer')# Simulate Homer's orderprint('...') scene.order('Shaggy')# Simulate Shaggy's order
c:\code> python pizzashop.py
Homer orders from <Employee: name=Pat, salary=40000>
Bob makes pizza
oven bakes
Homer pays for item to <Employee: name=Pat, salary=40000>
...
Shaggy orders from <Employee: name=Pat, salary=40000>
Bob makes pizza
oven bakes
Shaggy pays for item to <Employee: name=Pat, salary=40000>def processor(reader, converter, writer):
while True:
data = reader.read()
if not data: break
data = converter(data)
writer.write(data)class Processor:
def __init__(self, reader, writer):
self.reader = reader
self.writer = writer
def process(self):
while True:
data = self.reader.readline()
if not data: break
data = self.converter(data)
self.writer.write(data)
def converter(self, data):
assert False, 'converter must be defined' # Or raise exceptionfrom streams import Processor
class Uppercase(Processor):
def converter(self, data):
return data.upper()
if __name__ == '__main__':
import sys
obj = Uppercase(open('trispam.txt'), sys.stdout)
obj.process()c:\code>type trispam.txtspam Spam SPAM! c:\code>python converters.pySPAM SPAM SPAM!
C:\code>python>>>import converters>>>prog = converters.Uppercase(open('trispam.txt'), open('trispamup.txt', 'w'))>>>prog.process()C:\code>type trispamup.txtSPAM SPAM SPAM!
C:\code>python>>>from converters import Uppercase>>> >>>class HTMLize:def write(self, line):print('<PRE>%s</PRE>' % line.rstrip())>>>Uppercase(open('trispam.txt'), HTMLize()).process()<PRE>SPAM</PRE> <PRE>SPAM</PRE> <PRE>SPAM!</PRE>
class Wrapper:
def __init__(self, object):
self.wrapped = object # Save object
def __getattr__(self, attrname):
print('Trace: ' + attrname) # Trace fetch
return getattr(self.wrapped, attrname) # Delegate fetch>>>from trace import Wrapper>>>x = Wrapper([1, 2, 3])# Wrap a list>>>x.append(4)# Delegate to list methodTrace: append >>>x.wrapped# Print my member[1, 2, 3, 4] >>>x = Wrapper({'a': 1, 'b': 2})# Wrap a dictionary>>>list(x.keys())# Delegate to dictionary methodTrace: keys ['a', 'b']
class C1:
def meth1(self): self.X = 88 # I assume X is mine
def meth2(self): print(self.X)class C2:
def metha(self): self.X = 99 # Me too
def methb(self): print(self.X)class C3(C1, C2): ...
I = C3() # Only 1 X in I!class C1:
def meth1(self): self.__X = 88 # Now X is mine
def meth2(self): print(self.__X) # Becomes _C1__X in I
class C2:
def metha(self): self.__X = 99 # Me too
def methb(self): print(self.__X) # Becomes _C2__X in I
class C3(C1, C2): pass
I = C3() # Two X names in I
I.meth1(); I.metha()
print(I.__dict__)
I.meth2(); I.methb()% python pseudoprivate.py
{'_C2__X': 99, '_C1__X': 88}
88
99class Super:
def method(self): ... # A real application method
class Tool:
def __method(self): ... # Becomes _Tool__method
def other(self): self.__method() # Use my internal method
class Sub1(Tool, Super): ...
def actions(self): self.method() # Runs Super.method as expected
class Sub2(Tool):
def __init__(self): self.method = 99 # Doesn't break Tool.__methodselfAccessing a function attribute of a class by qualifying the class returns an unbound method object. To call the method, you must provide an instance object explicitly as the first argument. In Python 3.X, an unbound method is the same as a simple function and can be called through the class’s name; in 2.X it’s a distinct type and cannot be called without providing an instance.
self + function pairsAccessing a function attribute of a class by qualifying an instance returns a bound method object. Python automatically packages the instance with the function in the bound method object, so you don’t need to pass an instance to call the method.
class Spam:
def doit(self, message):
print(message)object1 = Spam()
object1.doit('hello world')object1 = Spam() x = object1.doit# Bound method object: instance+functionx('hello world')# Same effect as object1.doit('...')
object1 = Spam() t = Spam.doit# Unbound method object (a function in 3.X: see ahead)t(object1, 'howdy')# Pass in instance (if the method expects one in 3.X)
class Eggs:
def m1(self, n):
print(n)
def m2(self):
x = self.m1 # Another bound method object
x(42) # Looks like a simple function
Eggs().m2() # Prints 42C:\code>c:\python33\python>>>class Selfless:def __init__(self, data):self.data = datadef selfless(arg1, arg2):# A simple function in 3.Xreturn arg1 + arg2def normal(self, arg1, arg2):# Instance expected when calledreturn self.data + arg1 + arg2>>>X = Selfless(2)>>>X.normal(3, 4)# Instance passed to self automatically: 2+(3+4)9 >>>Selfless.normal(X, 3, 4)# self expected by method: pass manually9 >>>Selfless.selfless(3, 4)# No instance: works in 3.X, fails in 2.X!7
>>>X.selfless(3, 4)TypeError: selfless() takes 2 positional arguments but 3 were given >>>Selfless.normal(3, 4)TypeError: normal() missing 1 required positional argument: 'arg2'
>>>class Number:def __init__(self, base):self.base = basedef double(self):return self.base * 2def triple(self):return self.base * 3>>>x = Number(2)# Class instance objects>>>y = Number(3)# State + methods>>>z = Number(4)>>>x.double()# Normal immediate calls4 >>>acts = [x.double, y.double, y.triple, z.double]# List of bound methods>>>for act in acts:# Calls are deferredprint(act())# Call as though functions4 6 9 8
>>>bound = x.double>>>bound.__self__, bound.__func__(<__main__.Number object at 0x...etc...>, <function Number.double at 0x...etc...>) >>>bound.__self__.base2 >>>bound()# Calls bound.__func__(bound.__self__, ...)4
>>>def square(arg):return arg ** 2# Simple functions (def or lambda)>>>class Sum:def __init__(self, val):# Callable instancesself.val = valdef __call__(self, arg):return self.val + arg>>>class Product:def __init__(self, val):# Bound methodsself.val = valdef method(self, arg):return self.val * arg>>>sobject = Sum(2)>>>pobject = Product(3)>>>actions = [square, sobject, pobject.method]# Function, instance, method>>>for act in actions:# All three called same wayprint(act(5))# Call any one-arg callable25 7 15 >>>actions[-1](5)# Index, comprehensions, maps15 >>>[act(5) for act in actions][25, 7, 15] >>>list(map(lambda act: act(5), actions))[25, 7, 15]
>>>class Negate:def __init__(self, val):# Classes are callables tooself.val = -val# But called for object, not workdef __repr__(self):# Instance print formatreturn str(self.val)>>>actions = [square, sobject, pobject.method, Negate]# Call a class too>>>for act in actions:print(act(5))25 7 15 -5 >>>[act(5) for act in actions]# Runs __repr__ not __str__![25, 7, 15, −5] >>>table = {act(5): act for act in actions}# 3.X/2.7 dict comprehension>>>for (key, value) in table.items():print('{0:2} => {1}'.format(key, value))# 2.6+/3.X str.format25 => <function square at 0x0000000002987400> 15 => <bound method Product.method of <__main__.Product object at...etc...>> -5 => <class '__main__.Negate'> 7 => <__main__.Sum object at 0x000000000298BE48>
def factory(aClass, *pargs, **kargs):# Varargs tuple, dictreturn aClass(*pargs, **kargs)# Call aClass (or apply in 2.X only)class Spam: def doit(self, message): print(message) class Person: def __init__(self, name, job=None): self.name = name self.job = job object1 = factory(Spam)# Make a Spam objectobject2 = factory(Person, "Arthur", "King")# Make a Person objectobject3 = factory(Person, name='Brian')# Ditto, with keywords and default
>>>object1.doit(99)99 >>>object2.name, object2.job('Arthur', 'King') >>>object3.name, object3.job('Brian', None)
classname =...parse from config file...classarg =...parse from config file...import streamtypes# Customizable codeaclass = getattr(streamtypes, classname)# Fetch from modulereader = factory(aclass, classarg)# Or aclass(classarg)processor(reader, ...)
self.method(), for example. In this mode,
Python chooses the lowest and leftmost in classic classes, and in
nondiamond patterns in all classes; new-style classes may choose an
option to the right before one above in diamonds.superclass.method(self), for instance. Your
code breaks the conflict and overrides the search’s default — to select
an option to the right of or above the inheritance search’s
default.>>>class Spam:def __init__(self):# No __repr__ or __str__self.data1 = "food">>>X = Spam()>>>print(X)# Default: class name + address (id)<__main__.Spam object at 0x00000000029CA908># Same in 2.X, but says "instance"
AttrDisplay class formatted instance
attributes in a generic __repr__
method, but it did not climb class trees and was utilized in
single-inheritance mode only.#!python
# File listinstance.py (2.X + 3.X)
class ListInstance:
"""
Mix-in class that provides a formatted print() or str() of instances via
inheritance of __str__ coded here; displays instance attrs only; self is
instance of lowest class; __X names avoid clashing with client's attrs
"""
def __attrnames(self):
result = ''
for attr in sorted(self.__dict__):
result += '\t%s=%s\n' % (attr, self.__dict__[attr])
return result
def __str__(self):
return '<Instance of %s, address %s:\n%s>' % (
self.__class__.__name__, # My class's name
id(self), # My address
self.__attrnames()) # name=value list
if __name__ == '__main__':
import testmixin
testmixin.tester(ListInstance) def __attrnames(self):
return ''.join('\t%s=%s\n' % (attr, self.__dict__ [attr])
for attr in sorted(self.__dict__))__class__ attribute that references the
class from which it was created, and each class has a __name__
attribute that references the name in the header, so the
expression self.__class__.__name__ fetches the name
of an instance’s class.__dict__) to build up a string
showing the names and values of all instance attributes. The
dictionary’s keys are sorted to finesse any ordering differences
across Python releases.id built-in
function, which returns any object’s address (by definition, a
unique object identifier, which will be useful in later mutations
of this code).__attrnames. As we learned earlier in
this chapter, Python automatically localizes any such name to its
enclosing class by expanding the attribute name to include the
class name (in this case, it becomes _ListInstance__attrnames). This holds
true for both class attributes (like methods) and instance
attributes attached to self. As
noted in Chapter 28’s first-cut
version, this behavior is useful in a general tool like this, as
it ensures that its names don’t clash with any names used in its
client subclasses.>>>from listinstance import ListInstance>>>class Spam(ListInstance):# Inherit a __str__ methoddef __init__(self):self.data1 = 'food'>>>x = Spam()>>>print(x)# print() and str() run __str__<Instance of Spam, address 43034496: data1=food >
>>>display = str(x)# Print this to interpret escapes>>>display'<Instance of Spam, address 43034496:\n\tdata1=food\n>' >>>x# The __repr__ still is a default<__main__.Spam object at 0x000000000290A780>
# File testmixin0.py from listinstance import ListInstance# Get lister tool classclass Super: def __init__(self):# Superclass __init__self.data1 = 'spam'# Create instance attrsdef ham(self): pass class Sub(Super, ListInstance):# Mix in ham and a __str__def __init__(self):# Listers have access to selfSuper.__init__(self) self.data2 = 'eggs'# More instance attrsself.data3 = 42 def spam(self):# Define another method herepass if __name__ == '__main__': X = Sub() print(X)# Run mixed-in __str__
c:\code> python testmixin0.py
<Instance of Sub, address 44304144:
data1=spam
data2=eggs
data3=42
>#!python
# File testmixin.py (2.X + 3.X)
"""
Generic lister mixin tester: similar to transitive reloader in
Chapter 25, but passes a class object to tester (not function),
and testByNames adds loading of both module and class by name
strings here, in keeping with Chapter 31's factories pattern.
"""
import importlib
def tester(listerclass, sept=False):
class Super:
def __init__(self): # Superclass __init__
self.data1 = 'spam' # Create instance attrs
def ham(self):
pass
class Sub(Super, listerclass): # Mix in ham and a __str__
def __init__(self): # Listers have access to self
Super.__init__(self)
self.data2 = 'eggs' # More instance attrs
self.data3 = 42
def spam(self): # Define another method here
pass
instance = Sub() # Return instance with lister's __str__
print(instance) # Run mixed-in __str__ (or via str(x))
if sept: print('-' * 80)
def testByNames(modname, classname, sept=False):
modobject = importlib.import_module(modname) # Import by namestring
listerclass = getattr(modobject, classname) # Fetch attr by namestring
tester(listerclass, sept)
if __name__ == '__main__':
testByNames('listinstance', 'ListInstance', True) # Test all three here
testByNames('listinherited', 'ListInherited', True)
testByNames('listtree', 'ListTree', False)c:\code>python listinstance.py<Instance of Sub, address 43256968: data1=spam data2=eggs data3=42 > c:\code>python testmixin.py<Instance of Sub, address 43977584: data1=spam data2=eggs data3=42 >...and tests of two other lister classes coming up...
>>>import listinstance>>>class C(listinstance.ListInstance): pass>>>x = C()>>>x.a, x.b, x.c = 1, 2, 3>>>print(x)<Instance of C, address 43230824: a=1 b=2 c=3 >
#!python
# File listinherited.py (2.X + 3.X)
class ListInherited:
"""
Use dir() to collect both instance attrs and names inherited from
its classes; Python 3.X shows more names than 2.X because of the
implied object superclass in the new-style class model; getattr()
fetches inherited names not in self.__dict__; use __str__, not
__repr__, or else this loops when printing bound methods!
"""
def __attrnames(self):
result = ''
for attr in dir(self): # Instance dir()
if attr[:2] == '__' and attr[-2:] == '__': # Skip internals
result += '\t%s\n' % attr
else:
result += '\t%s=%s\n' % (attr, getattr(self, attr))
return result
def __str__(self):
return '<Instance of %s, address %s:\n%s>' % (
self.__class__.__name__, # My class's name
id(self), # My address
self.__attrnames()) # name=value list
if __name__ == '__main__':
import testmixin
testmixin.tester(ListInherited)c:\code>c:\python27\python listinherited.py<Instance of Sub, address 35161352: _ListInherited__attrnames=<bound method Sub.__attrnames of <test...more...>> __doc__ __init__ __module__ __str__ data1=spam data2=eggs data3=42 ham=<bound method Sub.ham of <testmixin.Sub instance at 0x00000...more...>> spam=<bound method Sub.spam of <testmixin.Sub instance at 0x00000...more...>> >
c:\code>c:\python33\python listinherited.py<Instance of Sub, address 43253152: _ListInherited__attrnames=<bound method Sub.__attrnames of <test...more...>> __class__ __delattr__ __dict__ __dir__ __doc__ __eq__...more names omitted 32 total...__repr__ __setattr__ __sizeof__ __str__ __subclasshook__ __weakref__ data1=spam data2=eggs data3=42 ham=<bound method Sub.ham of <testmixin.tester.<locals>.Sub ...more...>> spam=<bound method Sub.spam of <testmixin.tester.<locals>.Sub ...more...>> >
def __attrnames(self, indent=' '*4):
result = 'Unders%s\n%s%%s\nOthers%s\n' % ('-'*77, indent, '-'*77)
unders = []
for attr in dir(self): # Instance dir()
if attr[:2] == '__' and attr[-2:] == '__': # Skip internals
unders.append(attr)
else:
display = str(getattr(self, attr))[:82-(len(indent) + len(attr))]
result += '%s%s=%s\n' % (indent, attr, display)
return result % ', '.join(unders)c:\code>c:\python27\python listinherited2.py<Instance of Sub, address 36299208: Unders----------------------------------------------------------------------------- __doc__, __init__, __module__, __str__ Others----------------------------------------------------------------------------- _ListInherited__attrnames=<bound method Sub.__attrnames of <testmixin.Sub insta data1=spam data2=eggs data3=42 ham=<bound method Sub.ham of <testmixin.Sub instance at 0x000000000229E1C8>> spam=<bound method Sub.spam of <testmixin.Sub instance at 0x000000000229E1C8>> > c:\code>c:\python33\python listinherited2.py<Instance of Sub, address 43318912: Unders----------------------------------------------------------------------------- __class__, __delattr__, __dict__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __gt__, __hash__, __init__, __le__, __lt__, __module__, __ne__, __new__, __qualname__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__, __weakref__ Others----------------------------------------------------------------------------- _ListInherited__attrnames=<bound method Sub.__attrnames of <testmixin.tester.<l data1=spam data2=eggs data3=42 ham=<bound method Sub.ham of <testmixin.tester.<locals>.Sub object at 0x0000000 spam=<bound method Sub.spam of <testmixin.tester.<locals>.Sub object at 0x00000 >
#!python
# File listtree.py (2.X + 3.X)
class ListTree:
"""
Mix-in that returns an __str__ trace of the entire class tree and all
its objects' attrs at and above self; run by print(), str() returns
constructed string; uses __X attr names to avoid impacting clients;
recurses to superclasses explicitly, uses str.format() for clarity;
"""
def __attrnames(self, obj, indent):
spaces = ' ' * (indent + 1)
result = ''
for attr in sorted(obj.__dict__):
if attr.startswith('__') and attr.endswith('__'):
result += spaces + '{0}\n'.format(attr)
else:
result += spaces + '{0}={1}\n'.format(attr, getattr(obj, attr))
return result
def __listclass(self, aClass, indent):
dots = '.' * indent
if aClass in self.__visited:
return '\n{0}<Class {1}:, address {2}: (see above)>\n'.format(
dots,
aClass.__name__,
id(aClass))
else:
self.__visited[aClass] = True
here = self.__attrnames(aClass, indent)
above = ''
for super in aClass.__bases__:
above += self.__listclass(super, indent+4)
return '\n{0}<Class {1}, address {2}:\n{3}{4}{5}>\n'.format(
dots,
aClass.__name__,
id(aClass),
here, above,
dots)
def __str__(self):
self.__visited = {}
here = self.__attrnames(self, 0)
above = self.__listclass(self.__class__, 4)
return '<Instance of {0}, address {1}:\n{2}{3}>'.format(
self.__class__.__name__,
id(self),
here, above)
if __name__ == '__main__':
import testmixin
testmixin.tester(ListTree) above = ''
for super in aClass.__bases__:
above += self.__listclass(super, indent+4)
...or...
above = ''.join(
self.__listclass(super, indent+4) for super in aClass.__bases__) self.__visited[aClass] = True
genabove = (self.__listclass(c, indent+4) for c in aClass.__bases__)
return '\n{0}<Class {1}, address {2}:\n{3}{4}{5}>\n'.format(
dots,
aClass.__name__,
id(aClass),
self.__attrnames(aClass, indent), # Runs before format!
''.join(genabove),
dots)return '<Instance of %s, address %s:\n%s%s>' % (...)# Expressionreturn '<Instance of {0}, address {1}:\n{2}{3}>'.format(...)# Method
c:\code> c:\python27\python listtree.py
<Instance of Sub, address 36690632:
_ListTree__visited={}
data1=spam
data2=eggs
data3=42
....<Class Sub, address 36652616:
__doc__
__init__
__module__
spam=<unbound method Sub.spam>
........<Class Super, address 36652712:
__doc__
__init__
__module__
ham=<unbound method Super.ham>
........>
........<Class ListTree, address 30795816:
_ListTree__attrnames=<unbound method ListTree.__attrnames>
_ListTree__listclass=<unbound method ListTree.__listclass>
__doc__
__module__
__str__
........>
....>
>c:\code>c:\python33\python listtree.py<Instance of Sub, address 44277488: _ListTree__visited={} data1=spam data2=eggs data3=42 ....<Class Sub, address 36990264: __doc__ __init__ __module__ __qualname__ spam=<function tester.<locals>.Sub.spam at 0x0000000002A3C840> ........<Class Super, address 36989352: __dict__ __doc__ __init__ __module__ __qualname__ __weakref__ ham=<function tester.<locals>.Super.ham at 0x0000000002A3C730> ............<Class object, address 506770624: __class__ __delattr__ __dir__ __doc__ __eq__...more omitted: 22 total...__repr__ __setattr__ __sizeof__ __str__ __subclasshook__ ............> ........> ........<Class ListTree, address 36988440: _ListTree__attrnames=<function ListTree.__attrnames at 0x0000000002A3C158> _ListTree__listclass=<function ListTree.__listclass at 0x0000000002A3C1E0> __dict__ __doc__ __module__ __qualname__ __str__ __weakref__ ............<Class object:, address 506770624: (see above)> ........> ....> >
>>>class C: pass>>>class B(C): pass>>>C.__bases__ = (B,)# Deep, dark magic!TypeError: a __bases__ item causes an inheritance cycle
for attr in sorted(obj.__dict__):
# if attr.startswith('__') and attr.endswith('__'):
# result += spaces + '{0}\n'.format(attr)
# else:
result += spaces + '{0}={1}\n'.format(attr, getattr(obj, attr))c:\code> c:\python27\python listtree.py
<Instance of Sub, address 35750408:
_ListTree__visited={}
data1=spam
data2=eggs
data3=42
....<Class Sub, address 36353608:
__doc__=None
__init__=<unbound method Sub.__init__>
__module__=testmixin
spam=<unbound method Sub.spam>
........<Class Super, address 36353704:
__doc__=None
__init__=<unbound method Super.__init__>
__module__=testmixin
ham=<unbound method Super.ham>
........>
........<Class ListTree, address 31254568:
_ListTree__attrnames=<unbound method ListTree.__attrnames>
_ListTree__listclass=<unbound method ListTree.__listclass>
__doc__=
Mix-in that returns an __str__ trace of the entire class tree and all
its objects' attrs at and above self; run by print(), str() returns
constructed string; uses __X attr names to avoid impacting clients;
recurses to superclasses explicitly, uses str.format() for clarity;
__module__=__main__
__str__=<unbound method ListTree.__str__>
........>
....>
>c:\code>c:\python33\python listtree.py...etc...File "listtree.py", line 18, in __attrnames result += spaces + '{0}={1}\n'.format(attr, getattr(obj, attr)) TypeError: Type method_descriptor doesn't define __format__
c:\code>py −3.1>>>'{0}'.format(object.__reduce__)"<method '__reduce__' of 'object' objects>" c:\code>py −3.3>>>'{0}'.format(object.__reduce__)TypeError: Type method_descriptor doesn't define __format__
c:\code>py −3.3>>>'{0:s}'.format(object.__reduce__)TypeError: Type method_descriptor doesn't define __format__ >>>'{0!s}'.format(object.__reduce__)"<method '__reduce__' of 'object' objects>" >>>'{0}'.format(str(object.__reduce__))"<method '__reduce__' of 'object' objects>"
c:\code>py −3.3>>>'%s' % object.__reduce__ "<method '__reduce__' of 'object' objects>"
result += spaces + '{0}={1}\n'.format(attr, getattr(obj, attr))
result += spaces + '%s=%s\n' % (attr, getattr(obj, attr))>>>from listtree import ListTree>>>from tkinter import Button# Both classes have a __str__>>>class MyButton(ListTree, Button): pass# ListTree first: use its __str__>>>B = MyButton(text='spam')>>>open('savetree.txt', 'w').write(str(B))# Save to a file for later viewing20513 >>>len(open('savetree.txt').readlines())# Lines in the file330 >>>print(B)# Print the display here<Instance of MyButton, address 43363688: _ListTree__visited={} _name=43363688 _tclCommands=[] _w=.43363688 children={} master=....much more omitted...> >>>S = str(B)# Or print just the first part>>>print(S[:1000])
# File lister.py # Collect all three listers in one module for convenience from listinstance import ListInstance from listinherited import ListInherited from listtree import ListTree Lister = ListTree # Choose a default lister
>>>import lister>>>lister.ListInstance# Use a specific lister<class 'listinstance.ListInstance'> >>>lister.Lister# Use Lister default<class 'listtree.ListTree'> >>>from lister import Lister# Use Lister default>>>Lister<class 'listtree.ListTree'> >>>from lister import ListInstance as Lister# Use Lister alias>>>Lister<class 'listinstance.ListInstance'>
Grouping double-underscore names as we did earlier may help reduce the size of the tree display, though some like__init__are user-defined and may merit special treatment. Sketching the tree in a GUI might be a natural next step too — thetkintertoolkit that we utilized in the prior section’s lister examples ships with Python and provides basic but easy support, and others offer richer but more complex alternatives. See the notes at the end of Chapter 28’s case study for more pointers in this department.
In the next chapter, we’ll also meet the new-style class model, which modifies the search order for one special multiple inheritance case (diamonds). There, we’ll also study theclass.__mro__new-style class object attribute — a tuple giving the class tree search order used by inheritance, known as the new-style MRO.As is, ourListTreetree lister sketches the physical shape of the inheritance tree, and expects the viewer to infer from this where an attribute is inherited from. This was its goal, but a general object viewer might also use the MRO tuple to automatically associate an attribute with the class from which it is inherited — by scanning the new-style MRO (or the classic classes’ DFLR ordering) for each inherited attribute in adirresult, we can simulate Python’s inheritance search, and map attributes to their source objects in the physical class tree displayed.In fact, we will write code that comes very close to this idea in the next chapter’smapattrsmodule, and reuse this example’s test classes there to demonstrate the idea, so stay tuned for an epilogue to this story. This might be used instead of or in addition to displaying attribute physical locations in__attrnameshere; both forms might be useful data for programmers to see. This approach is also one way to deal with slots, the topic of the next note.
Because they scan instance__dict__namespace dictionaries, theListInstanceandListTreeclasses presented here raise some subtle design issues. In Python classes, some names associated with instance data may not be stored at the instance itself. This includes topics presented in the next chapter such as new-style properties, slots, and descriptors, but also attributes dynamically computed in all classes with tools like__getattr__. None of these “virtual” attributes’ names are stored in an instance’s namespace dictionary, so none will be displayed as part of an instance’s own data.Of these, slots seem the most strongly associated with an instance; they store data on instances, even though their names don’t appear in instance namespace dictionaries. Properties and descriptors are associated with instances too, but they don’t reserve space in the instance, their computed nature is much more explicit, and they may seem closer to class-level methods than instance data.As we’ll see in the next chapter, slots function like instance attributes, but are created and managed by automatically created items in classes. They are a relatively infrequently used new-style class option, where instance attributes are declared in a__slots__class attribute, and not physically stored in an instance’s__dict__; in fact, slots may suppress a__dict__entirely. Because of this, tools that display instances by scanning their namespaces alone won’t directly associate the instance with attributes stored in slots. As is,ListTreedisplays slots as class attributes wherever they appear (though not at the instance), andListInstancedoesn’t display them at all.Though this will make more sense after we study this feature in the next chapter, it impacts code here and similar tools. For example, if in textmixin.py we assign__slots__=['data1']inSuperand__slots__=['data3']inSub, only thedata2attribute is displayed in the instance by these two lister classes.ListTreealso displaysdata1anddata3, but as attributes of theSuperandSubclass objects and with a special format for their values (technically, they are class-level descriptors, another new-style tool introduced in the next chapter).As the next chapter will explain, to show slot attributes as instance names, tools generally need to usedirto get a list of all attributes — both physically present and inherited — and then use eithergetattrto fetch their values from the instance, or fetch values from their inheritance source via__dict__in tree scans and accept the display of the implementations of some at classes. Becausedirincludes the names of inherited “virtual” attributes — including both slots and properties — they would be included in the instance set. As we’ll also find, the MRO might assist here to mapdirattribute to their sources, or restrict instance displays to names coded in user-defined classes by filtering out names inherited from the built-inobject.ListInheritedis immune to most of this, because it already displays the fulldirresults set, which include both__dict__names and all classes’__slots__names, though its display is of marginal use as is. AListTreevariant using thedirtechnique along with the MRO sequence to map attributes to classes would apply to slots too, because slots-based names appear in class’s__dict__results individually as slot management tools, though not in the instance__dict__.Alternatively, as a policy we could simply let our code handle slot-based attributes as it currently does, rather than complicating it for a rarely used, advanced feature that’s even questionable practice today. Slots and normal instance attributes are different kinds of names. In fact, displaying slots names as attributes of classes instead of instances is technically more accurate — as we’ll see in the next chapter their implementation is at classes, though their space is at instances.Ultimately, attempting to collect all the “virtual” attributes associated with a class may be a bit of a pipe dream anyhow. Techniques like those outlined here may address slots and properties, but some attributes are entirely dynamic, with no physical basis at all: those computed on fetch by generic method such as__getattr__are not data in the classic sense. Tools that attempt to display data in a wildly dynamic language like Python must come with the caveat that some data is ethereal at best!3
class statement headers determines the
general order of attribute searches.__X) are used to
localize names to the enclosing class. This includes both class
attributes like methods defined inside the class, and self instance attributes assigned inside the
class’s methods. Such names are expanded to include the class name,
which makes them generally unique.1 This tends to scare people with a C++ background
disproportionately. In Python, it’s even possible to change or
completely delete a class’s method at runtime. On the other hand,
almost nobody ever does this in practical programs. As a scripting
language, Python is more about enabling than restricting. Also, recall
from our discussion of operator overloading in Chapter 30 that __getattr__ and __setattr__ can be used to emulate privacy,
but are generally not used for this purpose in practice. More on this
when we code a more realistic privacy decorator in Chapter 39.
2 Actually, this syntax can invoke any callable object, including
functions, classes, and methods. Hence, the factory function here can also run any
callable object, not just a class (despite the argument name). Also,
as we learned in Chapter 18, Python 2.X has an
alternative to aClass(*pargs,
**kargs): the apply(aClass, pargs,
kargs) built-in call, which has been removed in Python 3.X
because of its redundancy and limitations.
3 Some dynamic and proxy objects based on
__getattr__ and the like can also use the
__dir__ operator overloading method to
manually publish an attributes list for
dir calls. Because this is optional,
though, general tools cannot rely on their client classes to
do so. See Python
Pocket Reference, 5th Edition for more on the
__dir__ method.
class Set: def __init__(self, value = []):# Constructorself.data = []# Manages a listself.concat(value) def intersect(self, other):# other is any sequenceres = []# self is the subjectfor x in self.data: if x in other:# Pick common itemsres.append(x) return Set(res)# Return a new Setdef union(self, other):# other is any sequenceres = self.data[:]# Copy of my listfor x in other:# Add items in otherif not x in res: res.append(x) return Set(res) def concat(self, value):# value: list, Set...for x in value:# Removes duplicatesif not x in self.data: self.data.append(x) def __len__(self): return len(self.data)# len(self), if selfdef __getitem__(self, key): return self.data[key]# self[i], self[i:j]def __and__(self, other): return self.intersect(other)# self & otherdef __or__(self, other): return self.union(other)# self | otherdef __repr__(self): return 'Set:' + repr(self.data)# print(self),...def __iter__(self): return iter(self.data)# for x in self,...
from setwrapper import Set x = Set([1, 3, 5, 7]) print(x.union(Set([1, 4, 7])))# prints Set:[1, 3, 5, 7, 4]print(x | Set([1, 4, 6]))# prints Set:[1, 3, 5, 7, 4, 6]
# Subclass built-in list type/class# Map 1..N to 0..N-1; call back to built-in version.class MyList(list): def __getitem__(self, offset): print('(indexing %s at %s)' % (self, offset)) return list.__getitem__(self, offset - 1) if __name__ == '__main__': print(list('abc')) x = MyList('abc')# __init__ inherited from listprint(x)# __repr__ inherited from listprint(x[1])# MyList.__getitem__print(x[3])# Customizes list superclass methodx.append('spam'); print(x)# Attributes from list superclassx.reverse(); print(x)
% python typesubclass.py
['a', 'b', 'c']
['a', 'b', 'c']
(indexing ['a', 'b', 'c'] at 1)
a
(indexing ['a', 'b', 'c'] at 3)
c
['a', 'b', 'c', 'spam']
['spam', 'c', 'b', 'a']from __future__ import print_function# 2.X compatibilityclass Set(list): def __init__(self, value = []):# Constructorlist.__init__(self)# Customizes listself.concat(value)# Copies mutable defaultsdef intersect(self, other):# other is any sequenceres = []# self is the subjectfor x in self: if x in other:# Pick common itemsres.append(x) return Set(res)# Return a new Setdef union(self, other):# other is any sequenceres = Set(self)# Copy me and my listres.concat(other) return res def concat(self, value):# value: list, Set, etc.for x in value:# Removes duplicatesif not x in self: self.append(x) def __and__(self, other): return self.intersect(other) def __or__(self, other): return self.union(other) def __repr__(self): return 'Set:' + list.__repr__(self) if __name__ == '__main__': x = Set([1,3,5,7]) y = Set([2,1,4,5,6]) print(x, y, len(x)) print(x.intersect(y), y.union(x)) print(x & y, x | y) x.reverse(); print(x)
% python setsubclass.py
Set:[1, 3, 5, 7] Set:[2, 1, 4, 5, 6] 4
Set:[1, 5] Set:[2, 1, 4, 5, 6, 3, 7]
Set:[1, 5] Set:[1, 3, 5, 7, 2, 4, 6]
Set:[7, 5, 3, 1]object
or not. Coding the object
superclass is optional and implied.object (or another
built-in type) to be considered “new style” and enable and obtain all
new-style behavior. Classes without this are “classic.”class newstyle(object):# 2.X explicit new-style derivation...normal class code...# Not required in 3.X: automatic
The__getattr__and__getattribute__generic attribute interception methods are still run for attributes accessed by explicit name, but no longer for attributes implicitly fetched by built-in operations. They are not called for__X__operator overloading method names in built-in contexts only — the search for such names begins at classes, not instances. This breaks or complicates objects that serve as proxies for another object’s interface, if wrapped objects implement operator overloading. Such methods must be redefined for the sake of differing built-ins dispatch in new-style classes.
Classes are now types, and types are now classes. In fact, the two are essentially synonyms, though the metaclasses that now subsume types are still somewhat distinct from normal classes. Thetype(I)built-in returns the class an instance is made from, instead of a generic instance type, and is normally the same asI.__class__. Moreover, classes are instances of thetypeclass, andtypemay be subclassed to customize class creation with metaclasses coded withclassstatements. This can impact code that tests types or otherwise relies on the prior type model.
object root class:
defaultsAll new-style classes (and hence types) inherit fromobject, which comes with a small set of default operator overloading methods (e.g.,__repr__). In 3.X, this class is added automatically above the user-defined root (i.e., topmost) classes in a tree, and need not be listed as a superclass explicitly. This can affect code that assumes the absence of method defaults and root classes.
Diamond patterns of multiple inheritance have a slightly different search order — roughly, at diamonds they are searched across before up, and more breadth-first than depth-first. This attribute search order, known as the MRO, can be traced with a new__mro__attribute available on new-style classes. The new search order largely applies only to diamond class trees, though the new model’s impliedobjectroot itself forms a diamond in all multiple inheritance trees. Code that relies on the prior order will not work the same.
The algorithm used for inheritance in new-style classes is substantially more complex than the depth-first model of classic classes, incorporating special cases for descriptors, metaclasses, and built-ins. We won’t be able to formalize this until Chapter 40 after we’ve studied metaclasses and descriptors in more depth, but it can impact code that does not anticipate its extra convolutions.
New-style classes have a set of new class tools, including slots, properties, descriptors,super, and the__getattribute__method. Most of these have very specific tool-building purposes. Their use can also impact or break existing code, though; slots, for example, sometimes prevent creation of an instance namespace dictionary altogether, and generic attribute handlers may require different coding.
>>>class C:data = 'spam'def __getattr__(self, name):# Classic in 2.X: catches built-insprint(name)return getattr(self.data, name)>>>X = C()>>>X[0]__getitem__ 's' >>>print(X)# Classic doesn't inherit default__str__ spam >>>class C(object):# New-style in 2.X and 3.X...rest of class unchanged...>>>X = C()# Built-ins not routed to getattr>>>X[0]TypeError: 'C' object does not support indexing >>>print(X)<__main__.C object at 0x02205780>
>>>class C: pass# 2.X classic class>>>X = C()>>>X.normal = lambda: 99>>>X.normal()99 >>>X.__add__ = lambda(y): 88 + y>>>X.__add__(1)89 >>>X + 189 >>>class C(object): pass# 2.X/3.X new-style class>>>X = C()>>>X.normal = lambda: 99>>>X.normal()# Normals still from instance99 >>>X.__add__ = lambda(y): 88 + y>>>X.__add__(1)# Ditto for explicit built-in names89 >>>X + 1TypeError: unsupported operand type(s) for +: 'C' and 'int'
>>>class C(object):def __getattr__(self, name): print(name)>>>X = C()>>>X.normal# Normal names are still routed to getattrnormal >>>X.__add__# Direct calls by name are too, but expressions are not!__add__ >>>X + 1TypeError: unsupported operand type(s) for +: 'C' and 'int'
>>>class C(object):data = 'spam'def __getattr__(self, name):print('getattr: ' + name)return getattr(self.data, name)>>>X = C()>>>X.__getitem__(1)# Traditional mapping works but new-style's does notgetattr: __getitem__ 'p' >>>X[1]TypeError: 'C' object does not support indexing >>>type(X).__getitem__(X, 1)AttributeError: type object 'C' has no attribute '__getitem__' >>>X.__add__('eggs')# Ditto for +: instance skipped for expression onlygetattr: __add__ 'spameggs' >>>X + 'eggs'TypeError: unsupported operand type(s) for +: 'C' and 'str' >>>type(X).__add__(X, 'eggs')AttributeError: type object 'C' has no attribute '__add__'
>>>class C(object):# New-style: 3.X and 2.Xdata = 'spam'def __getattr__(self, name):# Catch normal namesprint('getattr: ' + name)return getattr(self.data, name)def __getitem__(self, i):# Redefine built-insprint('getitem: ' + str(i))return self.data[i]# Run expr or getattrdef __add__(self, other):print('add: ' + other)return getattr(self.data, '__add__')(other)>>>X = C()>>>X.uppergetattr: upper <built-in method upper of str object at 0x0233D670> >>>X.upper()getattr: upper 'SPAM' >>>X[1]# Built-in operation (implicit)getitem: 1 'p' >>>X.__getitem__(1)# Traditional equivalence (explicit)getitem: 1 'p' >>>type(X).__getitem__(X, 1)# New-style equivalencegetitem: 1 'p' >>>X + 'eggs'# Ditto for + and othersadd: eggs 'spameggs' >>>X.__add__('eggs')add: eggs 'spameggs' >>>type(X).__add__(X, 'eggs')add: eggs 'spameggs'
Thetypeobject generates classes as its instances, and classes generate instances of themselves. Both are considered types, because they generate instances. In fact, there is no real difference between built-in types like lists and strings and user-defined types coded as classes. This is why we can subclass built-in types, as shown earlier in this chapter — a subclass of a built-in type such aslistqualifies as a new-style class and becomes a new user-defined type.
New class-generating types may be coded in Python as the metaclasses we’ll meet later in this chapter — user-definedtypesubclasses that are coded with normalclassstatements, and control creation of the classes that are their instances. As we’ll see, metaclasses are both class and type, though they are distinct enough to support a reasonable argument that the prior type/class dichotomy has become one of metaclass/class, perhaps at the cost of added complexity in normal classes.
C:\code>c:\python27\python>>>class C: pass# Classic classes in 2.X>>>I = C()# Instances are made from classes>>>type(I), I.__class__(<type 'instance'>, <class __main__.C at 0x02399768>) >>>type(C)# But classes are not the same as types<type 'classobj'> >>>C.__class__AttributeError: class C has no attribute '__class__' >>>type([1, 2, 3]), [1, 2, 3].__class__(<type 'list'>, <type 'list'>) >>>type(list), list.__class__(<type 'type'>, <type 'type'>)
C:\code>c:\python27\python>>>class C(object): pass# New-style classes in 2.X>>>I = C()# Type of instance is class it's made from>>>type(I), I.__class__(<class '__main__.C'>, <class '__main__.C'>) >>>type(C), C.__class__# Classes are user-defined types(<type 'type'>, <type 'type'>)
C:\code>c:\python33\python>>>class C: pass>>>I = C()# All classes are new-style in 3.X>>>type(I), I.__class__# Type of instance is class it's made from(<class '__main__.C'>, <class '__main__.C'>) >>>type(C), C.__class__# Class is a type, and type is a class(<class 'type'>, <class 'type'>) >>>type([1, 2, 3]), [1, 2, 3].__class__(<class 'list'>, <class 'list'>) >>>type(list), list.__class__# Classes and built-in types work the same(<class 'type'>, <class 'type'>)
C:\code>c:\python33\python>>>class C: pass>>>class D: pass>>>c, d = C(), D()>>>type(c) == type(d)# 3.X: compares the instances' classesFalse >>>type(c), type(d)(<class '__main__.C'>, <class '__main__.D'>) >>>c.__class__, d.__class__(<class '__main__.C'>, <class '__main__.D'>) >>>c1, c2 = C(), C()>>>type(c1) == type(c2)True
C:\code>c:\python27\python>>>class C: pass>>>class D: pass>>>c, d = C(), D()>>>type(c) == type(d)# 2.X: all instances are same type!True >>>c.__class__ == d.__class__# Compare classes explicitly if neededFalse >>>type(c), type(d)(<type 'instance'>, <type 'instance'>) >>>c.__class__, d.__class__(<class __main__.C at 0x024585A0>, <class __main__.D at 0x024588D0>)
C:\code>c:\python27\python>>>class C(object): pass>>>class D(object): pass>>>c, d = C(), D()>>>type(c) == type(d)# 2.X new-style: same as all in 3.XFalse >>>type(c), type(d)(<class '__main__.C'>, <class '__main__.D'>) >>>c.__class__, d.__class__(<class '__main__.C'>, <class '__main__.D'>)
>>>class C: pass# For new-style classes>>>X = C()>>>type(X), type(C)# Type is class instance was created from(<class '__main__.C'>, <class 'type'>)
>>>isinstance(X, object)True >>>isinstance(C, object)# Classes always inherit from objectTrue
>>>type('spam'), type(str)(<class 'str'>, <class 'type'>) >>>isinstance('spam', object)# Same for built-in types (classes)True >>>isinstance(str, object)True
>>>type(type)# All classes are types, and vice versa<class 'type'> >>>type(object)<class 'type'> >>>isinstance(type, object)# All classes derive from object, even typeTrue >>>isinstance(object, type)# Types make classes, and type is a classTrue >>>type is objectFalse
c:\code>py −2>>>dir(object)['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__' , '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', ' __sizeof__', '__str__', '__subclasshook__'] >>>class C: pass>>>C.__bases__# Classic classes do not inherit from object() >>>X = C()>>>X.__repr__AttributeError: C instance has no attribute '__repr__' >>>class C(object): pass# New-style classes inherit object defaults>>>C.__bases__(<type 'object'>,) >>>X = C()>>>X.__repr__<method-wrapper '__repr__' of C object at 0x00000000020B5978> c:\code>py −3>>>class C: pass# This means all classes get defaults in 3.X>>>C.__bases__(<class 'object'>,) >>>C().__repr__<method-wrapper '__repr__' of C object at 0x0000000002955630>
The inheritance search path is more breadth-first in diamond cases — Python first looks in any superclasses to the right of the one just searched before ascending to the common superclass at the top. In other words, this search proceeds across by levels before moving up. This search order is called the new-style MRO for “method resolution order” (and often just MRO for short when used in contrast with the DFLR order). Despite the name, this is used for all attributes in Python, not just methods.
>>>class A:attr = 1# Classic (Python 2.X)>>>class B(A):pass# B and C both lead to A>>>class C(A):attr = 2>>>class D(B, C):pass# Tries A before C>>>x = D()>>>x.attr# Searches x, D, B, A1
>>>class A(object):attr = 1# New-style ("object" not required in 3.X)>>>class B(A): pass>>>class C(A): attr = 2>>>class D(B, C):pass# Tries C before A>>>x = D()>>>x.attr# Searches x, D, B, C2
>>>class A:attr = 1# Classic>>>class B(A): pass>>>class C(A): attr = 2>>>class D(B, C):attr = C.attr# <== Choose C, to the right>>>x = D()>>>x.attr# Works like new-style (all 3.X)2
>>>class A(object):attr = 1# New-style>>>class B(A):pass>>>class C(A): attr = 2>>>class D(B, C):attr = B.attr# <== Choose A.attr, above>>>x = D()>>>x.attr# Works like classic (default 2.X)1
>>>class A:def meth(s): print('A.meth')>>>class C(A):def meth(s): print('C.meth')>>>class B(A):pass>>>class D(B, C): pass# Use default search order>>>x = D()# Will vary per class type>>>x.meth()# Defaults to classic order in 2.XA.meth >>>class D(B, C): meth = C.meth# <== Pick C's method: new-style (and 3.X)>>>x = D()>>>x.meth()C.meth >>>class D(B, C): meth = B.meth# <== Pick B's method: classic>>>x = D()>>>x.meth()A.meth
class D(B, C):
def meth(self): # Redefine lower
...
C.meth(self) # <== Pick C's method by calling>>>class A: pass>>>class B(A): pass# Diamonds: order differs for newstyle>>>class C(A): pass# Breadth-first across lower levels>>>class D(B, C): pass>>>D.__mro__(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class 'object'>)
>>>class A: pass>>>class B(A): pass# Nondiamonds: order same as classic>>>class C: pass# Depth first, then left to right>>>class D(B, C): pass>>>D.__mro__(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.A'>, <class '__main__.C'>, <class 'object'>)
>>>class A: pass>>>class B: pass# Another nondiamond: DFLR>>>class C(A): pass>>>class D(B, C): pass>>>D.__mro__(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class 'object'>)
>>>A.__bases__# Superclass links: object at two roots(<class 'object'>,) >>>B.__bases__(<class 'object'>,) >>>C.__bases__(<class '__main__.A'>,) >>>D.__bases__(<class '__main__.B'>, <class '__main__.C'>)
>>>class X: pass>>>class Y: pass>>>class A(X): pass# Nondiamond: depth first then left to right>>>class B(Y): pass# Though implied "object" always forms a diamond>>>class D(A, B): pass>>>D.mro()[<class '__main__.D'>, <class '__main__.A'>, <class '__main__.X'>, <class '__main__.B'>, <class '__main__.Y'>, <class 'object'>] >>>X.__bases__, Y.__bases__((<class 'object'>,), (<class 'object'>,)) >>>A.__bases__, B.__bases__((<class '__main__.X'>,), (<class '__main__.Y'>,))
>>>D.mro() == list(D.__mro__)True >>>[cls.__name__ for cls in D.__mro__]['D', 'A', 'X', 'B', 'Y', 'object']
"""
File mapattrs.py (3.X + 2.X)
Main tool: mapattrs() maps all attributes on or inherited by an
instance to the instance or class from which they are inherited.
Assumes dir() gives all attributes of an instance. To simulate
inheritance, uses either the class's MRO tuple, which gives the
search order for new-style classes (and all in 3.X), or a recursive
traversal to infer the DFLR order of classic classes in 2.X.
Also here: inheritance() gives version-neutral class ordering;
assorted dictionary tools using 3.X/2.7 comprehensions.
"""
import pprint
def trace(X, label='', end='\n'):
print(label + pprint.pformat(X) + end) # Print nicely
def filterdictvals(D, V):
"""
dict D with entries for value V removed.
filterdictvals(dict(a=1, b=2, c=1), 1) => {'b': 2}
"""
return {K: V2 for (K, V2) in D.items() if V2 != V}
def invertdict(D):
"""
dict D with values changed to keys (grouped by values).
Values must all be hashable to work as dict/set keys.
invertdict(dict(a=1, b=2, c=1)) => {1: ['a', 'c'], 2: ['b']}
"""
def keysof(V):
return sorted(K for K in D.keys() if D[K] == V)
return {V: keysof(V) for V in set(D.values())}
def dflr(cls):
"""
Classic depth-first left-to-right order of class tree at cls.
Cycles not possible: Python disallows on __bases__ changes.
"""
here = [cls]
for sup in cls.__bases__:
here += dflr(sup)
return here
def inheritance(instance):
"""
Inheritance order sequence: new-style (MRO) or classic (DFLR)
"""
if hasattr(instance.__class__, '__mro__'):
return (instance,) + instance.__class__.__mro__
else:
return [instance] + dflr(instance.__class__)
def mapattrs(instance, withobject=False, bysource=False):
"""
dict with keys giving all inherited attributes of instance,
with values giving the object that each is inherited from.
withobject: False=remove object built-in class attributes.
bysource: True=group result by objects instead of attributes.
Supports classes with slots that preclude __dict__ in instances.
"""
attr2obj = {}
inherits = inheritance(instance)
for attr in dir(instance):
for obj in inherits:
if hasattr(obj, '__dict__') and attr in obj.__dict__: # See slots
attr2obj[attr] = obj
break
if not withobject:
attr2obj = filterdictvals(attr2obj, object)
return attr2obj if not bysource else invertdict(attr2obj)
if __name__ == '__main__':
print('Classic classes in 2.X, new-style in 3.X')
class A: attr1 = 1
class B(A): attr2 = 2
class C(A): attr1 = 3
class D(B, C): pass
I = D()
print('Py=>%s' % I.attr1) # Python's search == ours?
trace(inheritance(I), 'INH\n') # [Inheritance order]
trace(mapattrs(I), 'ATTRS\n') # Attrs => Source
trace(mapattrs(I, bysource=True), 'OBJS\n') # Source => [Attrs]
print('New-style classes in 2.X and 3.X')
class A(object): attr1 = 1 # "(object)" optional in 3.X
class B(A): attr2 = 2
class C(A): attr1 = 3
class D(B, C): pass
I = D()
print('Py=>%s' % I.attr1)
trace(inheritance(I), 'INH\n')
trace(mapattrs(I), 'ATTRS\n')
trace(mapattrs(I, bysource=True), 'OBJS\n')c:\code> py −2 mapattrs.py
Classic classes in 2.X, new-style in 3.X
Py=>1
INH
[<__main__.D instance at 0x000000000225A688>,
<class __main__.D at 0x0000000002248828>,
<class __main__.B at 0x0000000002248768>,
<class __main__.A at 0x0000000002248708>,
<class __main__.C at 0x00000000022487C8>,
<class __main__.A at 0x0000000002248708>]
ATTRS
{'__doc__': <class __main__.D at 0x0000000002248828>,
'__module__': <class __main__.D at 0x0000000002248828>,
'attr1': <class __main__.A at 0x0000000002248708>,
'attr2': <class __main__.B at 0x0000000002248768>}
OBJS
{<class __main__.A at 0x0000000002248708>: ['attr1'],
<class __main__.B at 0x0000000002248768>: ['attr2'],
<class __main__.D at 0x0000000002248828>: ['__doc__', '__module__']}
New-style classes in 2.X and 3.X
Py=>3
INH
(<__main__.D object at 0x0000000002257B38>,
<class '__main__.D'>,
<class '__main__.B'>,
<class '__main__.C'>,
<class '__main__.A'>,
<type 'object'>)
ATTRS
{'__dict__': <class '__main__.A'>,
'__doc__': <class '__main__.D'>,
'__module__': <class '__main__.D'>,
'__weakref__': <class '__main__.A'>,
'attr1': <class '__main__.C'>,
'attr2': <class '__main__.B'>}
OBJS
{<class '__main__.A'>: ['__dict__', '__weakref__'],
<class '__main__.B'>: ['attr2'],
<class '__main__.C'>: ['attr1'],
<class '__main__.D'>: ['__doc__', '__module__']}c:\code>py −3>>>from mapattrs import trace, dflr, inheritance, mapattrs>>>from testmixin0 import Sub>>>I = Sub()# Sub inherits from Super and ListInstance roots>>>trace(dflr(I.__class__))# 2.X search order: implied object before lister![<class 'testmixin0.Sub'>, <class 'testmixin0.Super'>, <class 'object'>, <class 'listinstance.ListInstance'>, <class 'object'>] >>>trace(inheritance(I))# 3.X (+ 2.X newstyle) search order: lister first(<testmixin0.Sub object at 0x0000000002974630>, <class 'testmixin0.Sub'>, <class 'testmixin0.Super'>, <class 'listinstance.ListInstance'>, <class 'object'>) >>>trace(mapattrs(I)){'_ListInstance__attrnames': <class 'listinstance.ListInstance'>, '__init__': <class 'testmixin0.Sub'>, '__str__': <class 'listinstance.ListInstance'>,...etc...'data1': <testmixin0.Sub object at 0x0000000002974630>, 'data2': <testmixin0.Sub object at 0x0000000002974630>, 'data3': <testmixin0.Sub object at 0x0000000002974630>, 'ham': <class 'testmixin0.Super'>, 'spam': <class 'testmixin0.Sub'>} >>>trace(mapattrs(I, bysource=True)){<testmixin0.Sub object at 0x0000000002974630>: ['data1', 'data2', 'data3'], <class 'listinstance.ListInstance'>: ['_ListInstance__attrnames', '__str__'], <class 'testmixin0.Super'>: ['__dict__', '__weakref__', 'ham'], <class 'testmixin0.Sub'>: ['__doc__', '__init__', '__module__', '__qualname__', 'spam']} >>>trace(mapattrs(I, withobject=True)){'_ListInstance__attrnames': <class 'listinstance.ListInstance'>, '__class__': <class 'object'>, '__delattr__': <class 'object'>,...etc...
>>>amap = mapattrs(I, withobject=True, bysource=True)>>>trace(amap){<testmixin0.Sub object at 0x0000000002974630>: ['data1', 'data2', 'data3'], <class 'listinstance.ListInstance'>: ['_ListInstance__attrnames', '__str__'], <class 'testmixin0.Super'>: ['__dict__', '__weakref__', 'ham'], <class 'testmixin0.Sub'>: ['__doc__', '__init__', '__module__', '__qualname__', 'spam'], <class 'object'>: ['__class__', '__delattr__',...etc...'__sizeof__', '__subclasshook__']}
# mapattrs-slots.py: test __slots__ attribute inheritancefrom mapattrs import mapattrs, trace class A(object): __slots__ = ['a', 'b']; x = 1; y = 2 class B(A): __slots__ = ['b', 'c'] class C(A): x = 2 class D(B, C): z = 3 def __init__(self): self.name = 'Bob'; I = D() trace(mapattrs(I, bysource=True))# Also: trace(mapattrs(I))
c:\code> py −3 mapattrs-slots.py
{<__main__.D object at 0x00000000028988E0>: ['name'],
<class '__main__.C'>: ['x'],
<class '__main__.D'>: ['__dict__',
'__doc__',
'__init__',
'__module__',
'__qualname__',
'__weakref__',
'z'],
<class '__main__.A'>: ['a', 'y'],
<class '__main__.B'>: ['__slots__', 'b', 'c']}>>>class limiter(object):__slots__ = ['age', 'name', 'job']>>>x = limiter()>>>x.age# Must assign before useAttributeError: age >>>x.age = 40# Looks like instance data>>>x.age40 >>>x.ape = 1000# Illegal: not in __slots__AttributeError: 'limiter' object has no attribute 'ape'
best reserved for rare cases where there are large numbers of instances in a memory-critical application.
>>>class C:# Requires "(object)" in 2.X only__slots__ = ['a', 'b']# __slots__ means no __dict__ by default>>>X = C()>>>X.a = 1>>>X.a1 >>>X.__dict__AttributeError: 'C' object has no attribute '__dict__'
>>>getattr(X, 'a')1 >>>setattr(X, 'b', 2)# But getattr() and setattr() still work>>>X.b2 >>>'a' in dir(X)# And dir() finds slot attributes tooTrue >>>'b' in dir(X)True
>>>class D:# Use D(object) for same result in 2.X__slots__ = ['a', 'b']def __init__(self):self.d = 4# Cannot add new names if no __dict__>>>X = D()AttributeError: 'D' object has no attribute 'd'
>>>class D:__slots__ = ['a', 'b', '__dict__']# Name __dict__ to include one tooc = 3# Class attrs work normallydef __init__(self):self.d = 4# d stored in __dict__, a is a slot>>>X = D()>>>X.d4 >>>X.c3 >>>X.a# All instance attrs undefined until assignedAttributeError: a >>>X.a = 1>>>X.b = 2
>>>X.__dict__# Some objects have both __dict__ and slot names{'d': 4}# getattr() can fetch either type of attr>>>X.__slots__['a', 'b', '__dict__'] >>>getattr(X, 'a'), getattr(X, 'c'), getattr(X, 'd')# Fetches all 3 forms(1, 3, 4)
>>>for attr in list(X.__dict__) + X.__slots__:# Wrong...print(attr, '=>', getattr(X, attr))
>>>for attr in list(getattr(X, '__dict__', [])) + getattr(X, '__slots__', []):print(attr, '=>', getattr(X, attr))d => 4 a => 1# Less wrong...b => 2 __dict__ => {'d': 4}
>>>class E:__slots__ = ['c', 'd']# Superclass has slots>>>class D(E):__slots__ = ['a', '__dict__']# But so does its subclass>>>X = D()>>>X.a = 1; X.b = 2; X.c = 3# The instance is the union (slots: a, c)>>>X.a, X.c(1, 3)
>>>E.__slots__# But slots are not concatenated['c', 'd'] >>>D.__slots__['a', '__dict__'] >>>X.__slots__# Instance inherits *lowest* __slots__['a', '__dict__'] >>>X.__dict__# And has its own an attr dict{'b': 2} >>>for attr in list(getattr(X, '__dict__', [])) + getattr(X, '__slots__', []):print(attr, '=>', getattr(X, attr))b => 2# Other superclass slots missed!a => 1 __dict__ => {'b': 2} >>>dir(X)# But dir() includes all slot names[...many names omitted... 'a', 'b', 'c', 'd']
>>>class Slotful:__slots__ = ['a', 'b', '__dict__']def __init__(self, data):self.c = data>>>I = Slotful(3)>>>I.a, I.b = 1, 2>>>I.a, I.b, I.c# Normal attribute fetch(1, 2, 3) >>>I.__dict__# Both __dict__ and slots storage{'c': 3} >>>[x for x in dir(I) if not x.startswith('__')]['a', 'b', 'c'] >>>I.__dict__['c']# __dict__ is only one attr source3 >>>getattr(I, 'c'), getattr(I, 'a')# dir+getattr is broader than __dict__(3, 1)# applies to slots, properties, descrip>>>for a in (x for x in dir(I) if not x.startswith('__')):print(a, getattr(I, a))a 1 b 2 c 3
__slots__, the
instance __dict__ attribute
created for the superclass will always be accessible, making a
__slots__ in the subclass
largely pointless. The subclass still manages its slots, but
doesn’t compute their values in any way, and doesn’t avoid a
dictionary — the main reason to use slots.__slots__ declaration is limited to the
class in which it appears, subclasses will produce an instance
__dict__ if they do not define
a __slots__, rendering a
__slots__ in a superclass
largely pointless.__slots__ preclude both an
instance __dict__ and assigning
names not listed, unless __dict__ is listed explicitly
too.>>>class C: pass# Bullet 1: slots in sub but not super>>>class D(C): __slots__ = ['a']# Makes instance dict for nonslots>>>X = D()# But slot name still managed in class>>>X.a = 1; X.b = 2>>>X.__dict__{'b': 2} >>>D.__dict__.keys()dict_keys([... 'a', '__slots__', ...]) >>>class C: __slots__ = ['a']# Bullet 2: slots in super but not sub>>>class D(C): pass# Makes instance dict for nonslots>>>X = D()# But slot name still managed in class>>>X.a = 1; X.b = 2>>>X.__dict__{'b': 2} >>>C.__dict__.keys()dict_keys([... 'a', '__slots__', ...]) >>>class C: __slots__ = ['a'] # Bullet 3: only lowest slot accessible>>>class D(C): __slots__ = ['a']>>>class C: __slots__ = ['a']; a = 99# Bullet 4: no class-level defaultsValueError: 'a' in __slots__ conflicts with class variable
>>>class C: __slots__ = ['a']# Assumes universal use, differing names>>>class D(C): __slots__ = ['b']>>>X = D()>>>X.a = 1; X.b = 2>>>X.__dict__AttributeError: 'D' object has no attribute '__dict__' >>>C.__dict__.keys(), D.__dict__.keys()(dict_keys([... 'a', '__slots__', ...]), dict_keys([... 'b', '__slots__', ...]))
class C(ListTree): pass X = C()# OK: no __slots__ usedprint(X) class C(ListTree): __slots__ = ['a', 'b']# OK: superclass produces __dict__X = C() X.c = 3 print(X)# Displays c at X, a and b at C
class A: __slots__ = ['a']# Both OK by bullet 1 aboveclass B(A, ListTree): pass class A: __slots__ = ['a'] class B(A, ListTree): __slots__ = ['b']# Displays b at B, a at A
def mapattrs(instance, withobject=False, bysource=False):
for attr in dir(instance):
for obj in inherits:
if attr in obj.__dict__: # May fail if __slots__ used
>>> class C: __slots__ = ['a']
>>> X = C()
>>> mapattrs(X)
AttributeError: 'C' object has no attribute '__dict__' if attr in getattr(obj, '__dict__', {}):
if hasattr(obj, '__dict__') and attr in obj.__dict__:# File slots-test.py
from __future__ import print_function
import timeit
base = """
Is = []
for i in range(1000):
X = C()
X.a = 1; X.b = 2; X.c = 3; X.d = 4
t = X.a + X.b + X.c + X.d
Is.append(X)
"""
stmt = """
class C(object):
__slots__ = ['a', 'b', 'c', 'd']
""" + base
print('Slots =>', end=' ')
print(min(timeit.repeat(stmt, number=1000, repeat=3)))
stmt = """
class C(object):
pass
""" + base
print('Nonslots=>', end=' ')
print(min(timeit.repeat(stmt, number=1000, repeat=3)))c:\code>py −3 slots-test.pySlots => 0.7780903942045899 Nonslots=> 0.9888108080898417 C:\code>py -2 slots-test.pySlots => 0.615521153591 Nonslots=> 0.766582559582
>>>class operators:def __getattr__(self, name):if name == 'age':return 40else:raise AttributeError(name)>>>x = operators()>>>x.age# Runs __getattr__40 >>>x.name# Runs __getattr__AttributeError: name
>>>class properties(object):# Need object in 2.X for settersdef getage(self):return 40age = property(getage, None, None, None)# (get, set, del, docs), or use @>>>x = properties()>>>x.age# Runs getage40 >>>x.name# Normal fetchAttributeError: 'properties' object has no attribute 'name'
>>>class properties(object):# Need object in 2.X for settersdef getage(self):return 40def setage(self, value):print('set age: %s' % value)self._age = valueage = property(getage, setage, None, None)>>>x = properties()>>>x.age# Runs getage40 >>>x.age = 42# Runs setageset age: 42 >>>x._age# Normal fetch: no getage call42 >>>x.age# Runs getage40 >>>x.job = 'trainer'# Normal assign: no setage call>>>x.job# Normal fetch: no getage call'trainer'
>>>class operators:def __getattr__(self, name):# On undefined referenceif name == 'age':return 40else:raise AttributeError(name)def __setattr__(self, name, value):# On all assignmentsprint('set: %s %s' % (name, value))if name == 'age':self.__dict__['_age'] = value# Or object.__setattr__()else:self.__dict__[name] = value>>>x = operators()>>>x.age# Runs __getattr__40 >>>x.age = 41# Runs __setattr__set: age 41 >>>x._age# Defined: no __getattr__ call41 >>>x.age# Runs __getattr__40 >>>x.job = 'trainer'# Runs __setattr__ againset: job trainer >>>x.job# Defined: no __getattr__ call'trainer'
class properties(object):
@property # Coding properties with decorators: ahead
def age(self):
...
@age.setter
def age(self, value):
...>>>class AgeDesc(object):def __get__(self, instance, owner): return 40def __set__(self, instance, value): instance._age = value>>>class descriptors(object):age = AgeDesc()>>>x = descriptors()>>>x.age# Runs AgeDesc.__get__40 >>>x.age = 42# Runs AgeDesc.__set__>>>x._age# Normal fetch: no AgeDesc call42
class Spam:
numInstances = 0
def __init__(self):
Spam.numInstances = Spam.numInstances + 1
def printNumInstances():
print("Number of instances created: %s" % Spam.numInstances)C:\code>c:\python27\python>>>from spam import Spam>>>a = Spam()# Cannot call unbound class methods in 2.X>>>b = Spam()# Methods expect a self object by default>>>c = Spam()>>>Spam.printNumInstances()TypeError: unbound method printNumInstances() must be called with Spam instance as first argument (got nothing instead) >>>a.printNumInstances()TypeError: printNumInstances() takes no arguments (1 given)
C:\code>c:\python33\python>>>from spam import Spam>>>a = Spam()# Can call functions in class in 3.X>>>b = Spam()# Calls through instances still pass a self>>>c = Spam()>>>Spam.printNumInstances()# Differs in 3.XNumber of instances created: 3 >>>a.printNumInstances()TypeError: printNumInstances() takes 0 positional arguments but 1 was given
Spam.printNumInstances()# Fails in 2.X, works in 3.Xinstance.printNumInstances()# Fails in both 2.X and 3.X (unless static)
def printNumInstances():
print("Number of instances created: %s" % Spam.numInstances)
class Spam:
numInstances = 0
def __init__(self):
Spam.numInstances = Spam.numInstances + 1
C:\code> c:\python33\python
>>> import spam
>>> a = spam.Spam()
>>> b = spam.Spam()
>>> c = spam.Spam()
>>> spam.printNumInstances() # But function may be too far removed
Number of instances created: 3 # And cannot be changed via inheritance
>>> spam.Spam.numInstances
3class Spam:
numInstances = 0
def __init__(self):
Spam.numInstances = Spam.numInstances + 1
def printNumInstances(self):
print("Number of instances created: %s" % Spam.numInstances)
C:\code> c:\python33\python
>>> from spam import Spam
>>> a, b, c = Spam(), Spam(), Spam()
>>> a.printNumInstances()
Number of instances created: 3
>>> Spam.printNumInstances(a)
Number of instances created: 3
>>> Spam().printNumInstances() # But fetching counter changes counter!
Number of instances created: 4# File bothmethods.pyclass Methods: def imeth(self, x):# Normal instance method: passed a selfprint([self, x]) def smeth(x):# Static: no instance passedprint([x]) def cmeth(cls, x):# Class: gets class, not instanceprint([cls, x]) smeth = staticmethod(smeth)# Make smeth a static method (or @: ahead)cmeth = classmethod(cmeth)# Make cmeth a class method (or @: ahead)
self instance object (the default)staticmethod)classmethod, and inherent in
metaclasses)>>>from bothmethods import Methods# Normal instance methods>>>obj = Methods()# Callable through instance or class>>>obj.imeth(1)[<bothmethods.Methods object at 0x0000000002A15710>, 1] >>>Methods.imeth(obj, 2)[<bothmethods.Methods object at 0x0000000002A15710>, 2]
>>>Methods.smeth(3)# Static method: call through class[3]# No instance passed or expected>>>obj.smeth(4)# Static method: call through instance[4]# Instance not passed
>>>Methods.cmeth(5)# Class method: call through class[<class 'bothmethods.Methods'>, 5]# Becomes cmeth(Methods, 5)>>>obj.cmeth(6)# Class method: call through instance[<class 'bothmethods.Methods'>, 6]# Becomes cmeth(Methods, 6)
class Spam:
numInstances = 0 # Use static method for class data
def __init__(self):
Spam.numInstances += 1
def printNumInstances():
print("Number of instances: %s" % Spam.numInstances)
printNumInstances = staticmethod(printNumInstances)>>>from spam_static import Spam>>>a = Spam()>>>b = Spam()>>>c = Spam()>>>Spam.printNumInstances()# Call as simple functionNumber of instances: 3 >>>a.printNumInstances()# Instance argument not passedNumber of instances: 3
class Sub(Spam):
def printNumInstances(): # Override a static method
print("Extra stuff...") # But call back to original
Spam.printNumInstances()
printNumInstances = staticmethod(printNumInstances)
>>> from spam_static import Spam, Sub
>>> a = Sub()
>>> b = Sub()
>>> a.printNumInstances() # Call from subclass instance
Extra stuff...
Number of instances: 2
>>> Sub.printNumInstances() # Call from subclass itself
Extra stuff...
Number of instances: 2
>>> Spam.printNumInstances() # Call original version
Number of instances: 2>>>class Other(Spam): pass# Inherit static method verbatim>>>c = Other()>>>c.printNumInstances()Number of instances: 3
class Spam:
numInstances = 0 # Use class method instead of static
def __init__(self):
Spam.numInstances += 1
def printNumInstances(cls):
print("Number of instances: %s" % cls.numInstances)
printNumInstances = classmethod(printNumInstances)>>>from spam_class import Spam>>>a, b = Spam(), Spam()>>>a.printNumInstances()# Passes class to first argumentNumber of instances: 2 >>>Spam.printNumInstances()# Also passes class to first argumentNumber of instances: 2
class Spam:
numInstances = 0 # Trace class passed in
def __init__(self):
Spam.numInstances += 1
def printNumInstances(cls):
print("Number of instances: %s %s" % (cls.numInstances, cls))
printNumInstances = classmethod(printNumInstances)
class Sub(Spam):
def printNumInstances(cls): # Override a class method
print("Extra stuff...", cls) # But call back to original
Spam.printNumInstances()
printNumInstances = classmethod(printNumInstances)
class Other(Spam): pass # Inherit class method verbatim>>>from spam_class import Spam, Sub, Other>>>x = Sub()>>>y = Spam()>>>x.printNumInstances()# Call from subclass instanceExtra stuff... <class 'spam_class.Sub'> Number of instances: 2 <class 'spam_class.Spam'> >>>Sub.printNumInstances()# Call from subclass itselfExtra stuff... <class 'spam_class.Sub'> Number of instances: 2 <class 'spam_class.Spam'> >>>y.printNumInstances()# Call from superclass instanceNumber of instances: 2 <class 'spam_class.Spam'>
>>>z = Other()# Call from lower sub's instance>>>z.printNumInstances()Number of instances: 3 <class 'spam_class.Other'>
class Spam:
numInstances = 0
def count(cls): # Per-class instance counters
cls.numInstances += 1 # cls is lowest class above instance
def __init__(self):
self.count() # Passes self.__class__ to count
count = classmethod(count)
class Sub(Spam):
numInstances = 0
def __init__(self): # Redefines __init__
Spam.__init__(self)
class Other(Spam): # Inherits __init__
numInstances = 0
>>> from spam_class2 import Spam, Sub, Other
>>> x = Spam()
>>> y1, y2 = Sub(), Sub()
>>> z1, z2, z3 = Other(), Other(), Other()
>>> x.numInstances, y1.numInstances, z1.numInstances # Per-class data!
(1, 2, 3)
>>> Spam.numInstances, Sub.numInstances, Other.numInstances
(1, 2, 3)class C:
@staticmethod # Function decoration syntax
def meth():
...class C:
def meth():
...
meth = staticmethod(meth) # Name rebinding equivalentclass Spam:
numInstances = 0
def __init__(self):
Spam.numInstances = Spam.numInstances + 1
@staticmethod
def printNumInstances():
print("Number of instances created: %s" % Spam.numInstances)
>>> from spam_static_deco import Spam
>>> a = Spam()
>>> b = Spam()
>>> c = Spam()
>>> Spam.printNumInstances() # Calls from classes and instances work
Number of instances created: 3
>>> a.printNumInstances()
Number of instances created: 3# File bothmethods_decorators.pyclass Methods(object):# object needed in 2.X for property settersdef imeth(self, x):# Normal instance method: passed a selfprint([self, x]) @staticmethod def smeth(x):# Static: no instance passedprint([x]) @classmethod def cmeth(cls, x):# Class: gets class, not instanceprint([cls, x]) @property# Property: computed on fetchdef name(self): return 'Bob ' + self.__class__.__name__ >>>from bothmethods_decorators import Methods>>>obj = Methods()>>>obj.imeth(1)[<bothmethods_decorators.Methods object at 0x0000000002A256A0>, 1] >>>obj.smeth(2)[2] >>>obj.cmeth(3)[<class 'bothmethods_decorators.Methods'>, 3] >>>obj.name'Bob Methods'
class tracer:
def __init__(self, func): # Remember original, init counter
self.calls = 0
self.func = func
def __call__(self, *args): # On later calls: add logic, run original
self.calls += 1
print('call %s to %s' % (self.calls, self.func.__name__))
return self.func(*args)
@tracer # Same as spam = tracer(spam)
def spam(a, b, c): # Wrap spam in a decorator object
return a + b + c
print(spam(1, 2, 3)) # Really calls the tracer wrapper object
print(spam('a', 'b', 'c')) # Invokes __call__ in classc:\code> python tracer1.py
call 1 to spam
6
call 2 to spam
abcdef tracer(func):# Remember originaldef oncall(*args):# On later callsoncall.calls += 1 print('call %s to %s' % (oncall.calls, func.__name__)) return func(*args) oncall.calls = 0 return oncall class C: @tracer def spam(self,a, b, c): return a + b + c x = C() print(x.spam(1, 2, 3)) print(x.spam('a', 'b', 'c'))# Same output as tracer1 (in tracer2.py)
def decorator(aClass): ...
@decorator # Class decoration syntax
class C: ...def decorator(aClass): ...
class C: ... # Name rebinding equivalent
C = decorator(C)def count(aClass):
aClass.numInstances = 0
return aClass # Return class itself, instead of a wrapper
@count
class Spam: ... # Same as Spam = count(Spam)
@count
class Sub(Spam): ... # numInstances = 0 not needed here
@count
class Other(Spam): ...@count def spam(): pass# Like spam = count(spam)@count class Other: pass# Like Other = count(Other)spam.numInstances# Both are set to zeroOther.numInstances
def decorator(cls):# On @ decorationclass Proxy: def __init__(self, *args):# On instance creation: make a clsself.wrapped = cls(*args) def __getattr__(self, name):# On attribute fetch: extra ops herereturn getattr(self.wrapped, name) return Proxy @decorator class C: ...# Like C = decorator(C)X = C()# Makes a Proxy that wraps a C, and catches later X.attr
class Meta(type):
def __new__(meta, classname, supers, classdict):
...extra logic + class creation via type call...
class C(metaclass=Meta):
...my creation routed to Meta... # Like C = Meta('C', (), {...})class C:
__metaclass__ = Meta
... my creation routed to Meta...classname= Meta(classname,superclasses,attributedict)
>>>class C:# In Python 2.X and 3.Xdef act(self):print('spam')>>>class D(C):def act(self):C.act(self)# Name superclass explicitly, pass selfprint('eggs')>>>X = D()>>>X.act()spam eggs
>>>class C:# In Python 3.X (only: see 2.X super form ahead)def act(self):print('spam')>>>class D(C):def act(self):super().act()# Reference superclass generically, omit selfprint('eggs')>>>X = D()>>>X.act()spam eggs
>>>super# A "magic" proxy object that routes later calls<class 'super'> >>>super()SystemError: super(): no arguments >>>class E(C):def method(self):# self is implicit in super...only!proxy = super()# This form has no meaning outside a methodprint(proxy)# Show the normally hidden proxy objectproxy.act()# No arguments: implicitly calls superclass method!>>>E().method()<super: <class 'E'>, <E object>> spam
>>>class A:# In Python 3.Xdef act(self): print('A')>>>class B:def act(self): print('B')>>>class C(A):def act(self):super().act()# super applied to a single-inheritance tree>>>X = C()>>>X.act()A
>>>class C(A, B):# Add a B mix-in class with the same methoddef act(self):super().act()# Doesn't fail on multi-inher, but picks just one!>>>X = C()>>>X.act()A >>>class C(B, A):def act(self):super().act()#If B is listed first, A.act() is no longer run!>>>X = C()>>>X.act()B
>>>class C(A, B):# Traditional formdef act(self):# You probably need to be more explicit hereA.act(self)# This form handles both single and multiple inherB.act(self)# And works the same in both Python 3.X and 2.X>>>X = C()# So why use the super() special case at all?>>>X.act()A B
class PyMailServerWindow(PyMailServer, windows.MainWindow):
"a Tk, with extra protocol and mixed-in methods"
def __init__(self):
windows.MainWindow.__init__(self, appname, srvrname)
PyMailServer.__init__(self)
class PyMailFileWindow(PyMailFile, windows.PopupWindow):
"a Toplevel, with extra protocol and mixed-in methods"
def __init__(self, filename):
windows.PopupWindow.__init__(self, appname, filename)
PyMailFile.__init__(self, filename)>>>class C:# In Python 3.Xdef __getitem__(self, ix):# Indexing overload methodprint('C index')>>>class D(C):def __getitem__(self, ix):# Redefine to extend hereprint('D index')C.__getitem__(self, ix)# Traditional call form workssuper().__getitem__(ix)# Direct name calls work toosuper()[ix]# But operators do not! (__getattribute__)>>>X = C()>>>X[99]C index >>>X = D()>>>X[99]D index C index C index Traceback (most recent call last): File "", line 1, in File "", line 6, in __getitem__ TypeError: 'super' object is not subscriptable
>>>class C(object):# In Python 2.X: for new-style classes onlydef act(self):print('spam')>>>class D(C):def act(self):super(D, self).act()# 2.X: different call format - seems too complexprint('eggs')# "D" may be just as much to type/change as "C"!>>>X = D()>>>X.act()spam eggs
>>>class D(C):def act(self):super().act()# Simpler 3.X call format fails in 2.Xprint('eggs')>>>X = D()>>>X.act()TypeError: super() takes at least 1 argument (0 given)
>>>class D(C):def act(self):C.act(self)# But traditional pattern works portablyprint('eggs')# And may often be simpler in 2.X code>>>X = D()>>>X.act()spam eggs
super.super can provide a protocol for orderly
call routing.super
in each version of the method in the tree to be effective. Such
dispatch can also often be implemented in other ways (e.g., via
instance state).>>>class X:def m(self): print('X.m')>>>class Y:def m(self): print('Y.m')>>>class C(X):# Start out inheriting from Xdef m(self): super().m()# Can't hardcode class name here>>>i = C()>>>i.m()X.m >>>C.__bases__ = (Y,)# Change superclass at runtime!>>>i.m()Y.m
>>>class C(X):def m(self): C.__bases__[0].m(self)# Special code for a special case>>>i = C()>>>i.m()X.m >>>C.__bases__ = (Y,)# Same effect, without super()>>>i.m()Y.m
>>>class B:def __init__(self): print('B.__init__')# Disjoint class tree branches>>>class C:def __init__(self): print('C.__init__')>>>class D(B, C): pass>>>x = D()# Runs leftmost only by defaultB.__init__
>>>class D(B, C):def __init__(self):# Traditional formB.__init__(self)# Invoke supers by nameC.__init__(self)>>>x = D()B.__init__ C.__init__
>>>class A:def __init__(self): print('A.__init__')>>>class B(A):def __init__(self): print('B.__init__'); A.__init__(self)>>>class C(A):def __init__(self): print('C.__init__'); A.__init__(self)>>>x = B()B.__init__ A.__init__ >>>x = C()# Each super works by itselfC.__init__ A.__init__ >>>class D(B, C): pass# Still runs leftmost only>>>x = D()B.__init__ A.__init__ >>>class D(B, C):def __init__(self):# Traditional formB.__init__(self)# Invoke both supers by nameC.__init__(self)>>>x = D()# But this now invokes A twice!B.__init__ A.__init__ C.__init__ A.__init__
>>>class A:def __init__(self): print('A.__init__')>>>class B(A):def __init__(self): print('B.__init__'); super().__init__()>>>class C(A):def __init__(self): print('C.__init__'); super().__init__()>>>x = B()# Runs B.__init__, A is next super in self's B MROB.__init__ A.__init__ >>>x = C()C.__init__ A.__init__ >>>class D(B, C): pass>>>x = D()# Runs B.__init__, C is next super in self's D MRO!B.__init__ C.__init__ A.__init__
>>>B.__mro__(<class '__main__.B'>, <class '__main__.A'>, <class 'object'>) >>>D.__mro__(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class 'object'>)
>>>class B:def __init__(self): print('B.__init__'); super().__init__()>>>class C:def __init__(self): print('C.__init__'); super().__init__()>>>x = B()# object is an implied super at the end of MROB.__init__ >>>x = C()C.__init__ >>>class D(B, C): pass# Inherits B.__init__ but B's MRO differs for D>>>x = D()# Runs B.__init__, C is next super in self's D MRO!B.__init__ C.__init__
>>>B.__mro__(<class '__main__.B'>, <class 'object'>) >>>D.__mro__(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class 'object'>)
>>>class B:def __init__(self): print('B.__init__')>>>class C:def __init__(self): print('C.__init__')>>>class D(B, C):def __init__(self): B.__init__(self); C.__init__(self)>>>x = D()B.__init__ C.__init__
>>>class B:def __init__(self): print('B.__init__'); super().__init__()>>>class C:def __init__(self): print('C.__init__'); super().__init__()>>>class D(B, C):def __init__(self): print('D.__init__'); super().__init__()>>>X = D()D.__init__ B.__init__ C.__init__ >>>D.__mro__(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class 'object'>)# What if you must use a class that doesn't call super?>>>class B:def __init__(self): print('B.__init__')>>>class D(B, C):def __init__(self): print('D.__init__'); super().__init__()>>>X = D()D.__init__ B.__init__# It's an all-or-nothing tool...
# What if method call ordering needs differ from the MRO?>>>class B:def __init__(self): print('B.__init__'); super().__init__()>>>class C:def __init__(self): print('C.__init__'); super().__init__()>>>class D(B, C):def __init__(self): print('D.__init__'); C.__init__(self); B.__init__(self)>>>X = D()D.__init__ C.__init__ B.__init__ C.__init__# It's the MRO xor explicit calls...
>>>class A:def method(self): print('A.method'); super().method()>>>class B(A):def method(self): print('B.method'); super().method()>>>class C:def method(self): print('C.method')# No super: must anchor the chain!>>>class D(B, C):def method(self): print('D.method'); super().method()>>>X = D()>>>X.method()D.method B.method A.method# Dispatch to all per the MRO automaticallyC.method
# What if a class needs to replace a super's default entirely?>>>class B(A):def method(self): print('B.method')# Drop super to replace A's method>>>class D(B, C):def method(self): print('D.method'); super().method()>>>X = D()>>>X.method()D.method B.method# But replacement also breaks the call chain...>>>class D(B, C):def method(self): print('D.method'); B.method(self); C.method(self)>>>D().method()D.method B.method C.method# It's back to explicit calls...
# Mix-ins work for disjoint method sets>>>class A:def other(self): print('A.other')>>>class Mixin(A):def other(self): print('Mixin.other'); super().other()>>>class B:def method(self): print('B.method')>>>class C(Mixin, B):def method(self): print('C.method'); super().other(); super().method()>>>C().method()C.method Mixin.other A.other B.method >>>C.__mro__(<class '__main__.C'>, <class '__main__.Mixin'>, <class '__main__.A'>, <class '__main__.B'>, <class 'object'>)
>>>class C(B, Mixin):def method(self): print('C.method'); super().other(); super().method()>>>C().method()C.method Mixin.other A.other B.method >>>C.__mro__(<class '__main__.C'>, <class '__main__.B'>, <class '__main__.Mixin'>, <class '__main__.A'>, <class 'object'>)
# Explicit diamonds work too>>>class A:def other(self): print('A.other')>>>class Mixin(A):def other(self): print('Mixin.other'); super().other()>>>class B(A):def method(self): print('B.method')>>>class C(Mixin, B):def method(self): print('C.method'); super().other(); super().method()>>>C().method()C.method Mixin.other A.other B.method >>>C.__mro__(<class '__main__.C'>, <class '__main__.Mixin'>, <class '__main__.B'>, <class '__main__.A'>, <class 'object'>)# Other mix-in orderings work too>>>class C(B, Mixin):def method(self): print('C.method'); super().other(); super().method()>>>C().method()C.method Mixin.other A.other B.method >>>C.__mro__(<class '__main__.C'>, <class '__main__.B'>, <class '__main__.Mixin'>, <class '__main__.A'>, <class 'object'>)
# But direct calls work here too: explicit is better than implicit>>>class C(Mixin, B):def method(self): print('C.method'); Mixin.other(self); B.method(self)>>>X = C()>>>X.method()C.method Mixin.other A.other B.method
# But for nondisjoint methods: super creates overly strong coupling>>>class A:def method(self): print('A.method')>>>class Mixin(A):def method(self): print('Mixin.method'); super().method()>>>Mixin().method()Mixin.method A.method >>>class B(A):def method(self): print('B.method')# super here would invoke A after B>>>class C(Mixin, B):def method(self): print('C.method'); super().method()>>>C().method()C.method Mixin.method B.method# We miss A in this context only!
# And direct calls do not: they are immune to context of use>>>class A:def method(self): print('A.method')>>>class Mixin(A):def method(self): print('Mixin.method'); A.method(self)# C irrelevant>>>class C(Mixin, B):def method(self): print('C.method'); Mixin.method(self)>>>C().method()C.method Mixin.method A.method
>>>class Employee:def __init__(self, name, salary):# Common superclassself.name = nameself.salary = salary>>>class Chef1(Employee):def __init__(self, name):# Differing argumentsEmployee.__init__(self, name, 50000)# Dispatch by direct call>>>class Server1(Employee):def __init__(self, name):Employee.__init__(self, name, 40000)>>>bob = Chef1('Bob')>>>sue = Server1('Sue')>>>bob.salary, sue.salary(50000, 40000)
>>>class Chef2(Employee):def __init__(self, name):super().__init__(name, 50000)# Dispatch by super()>>>class Server2(Employee):def __init__(self, name):super().__init__(name, 40000)>>>bob = Chef2('Bob')>>>sue = Server2('Sue')>>>bob.salary, sue.salary(50000, 40000)
>>>class TwoJobs(Chef2, Server2): pass>>>tom = TwoJobs('Tom')TypeError: __init__() takes 2 positional arguments but 3 were given
>>>TwoJobs.__mro__(<class '__main__.TwoJobs'>, <class '__main__.Chef2'>, <class '__main__.Server2'> <class '__main__.Employee'>, <class 'object'>) >>>Chef2.__mro__(<class '__main__.Chef2'>, <class '__main__.Employee'>, <class 'object'>)
>>>class TwoJobs(Chef1, Server1): pass>>>tom = TwoJobs('Tom')>>>tom.salary50000
>>>class TwoJobs(Chef1, Server1):def __init__(self, name): Employee.__init__(self, name, 70000)>>>tom = TwoJobs('Tom')>>>tom.salary70000 >>>class TwoJobs(Chef2, Server2):def __init__(self, name): super().__init__(name, 70000)>>>tom = TwoJobs('Tom')TypeError: __init__() takes 2 positional arguments but 3 were given
super must exist — which requires extra
code if no anchor is present.super must have the same argument
signature across the class tree — which impairs flexibility,
especially for implementation-level methods like
constructors.super but the last must use super itself — which makes it difficult to
use existing code, change call ordering, override methods, and
code self-contained classes.If the implementation is hard to explain, it’s a bad idea.
>>>class X:a = 1# Class attribute>>>I = X()>>>I.a# Inherited by instance1 >>>X.a1
>>>X.a = 2# May change more than X>>>I.a# I changes too2 >>>J = X()# J inherits from X's runtime values>>>J.a# (but assigning to J.a changes a in J, not X or I)2
class X: pass# Make a few attribute namespacesclass Y: pass X.a = 1# Use class attributes as variablesX.b = 2# No instances anywhere to be foundX.c = 3 Y.a = X.a + X.b + X.c for X.i in range(Y.a): print(X.i)# Prints 0..5
class Record: pass X = Record() X.name = 'bob' X.job = 'Pizza maker'
>>>class C:shared = []# Class attributedef __init__(self):self.perobj = []# Instance attribute>>>x = C()# Two instances>>>y = C()# Implicitly share class attrs>>>y.shared, y.perobj([], []) >>>x.shared.append('spam')# Impacts y's view too!>>>x.perobj.append('spam')# Impacts x's data only>>>x.shared, x.perobj(['spam'], ['spam']) >>>y.shared, y.perobj# y sees change made through x(['spam'], []) >>>C.shared# Stored on class and shared['spam']
x.shared.append('spam') # Changes shared object attached to class in place
x.shared = 'spam' # Changed or creates instance attribute attached to xclass ListTree:
def __str__(self): ...
class Super:
def __str__(self): ...
class Sub(ListTree, Super): # Get ListTree's __str__ by listing it first
x = Sub() # Inheritance searches ListTree before Superclass ListTree:
def __str__(self): ...
def other(self): ...
class Super:
def __str__(self): ...
def other(self): ...
class Sub(ListTree, Super): # Get ListTree's __str__ by listing it first
other = Super.other # But explicitly pick Super's version of other
def __init__(self):
...
x = Sub() # Inheritance searches Sub before ListTree/Superclass Sub(Super, ListTree):# Get Super's other by order__str__ = Lister.__str__# Explicitly pick Lister.__str__
def generate():
class Spam: # Spam is a name in generate's local scope
count = 1
def method(self):
print(Spam.count) # Visible in generate's scope, per LEGB rule (E)
return Spam()
generate().method()def generate():
return Spam()
class Spam: # Define at top level of module
count = 1
def method(self):
print(Spam.count) # Works: in global (enclosing module)
generate().method()>>>def generate(label):# Returns a class instead of an instanceclass Spam:count = 1def method(self):print("%s=%s" % (label, Spam.count))return Spam>>>aclass = generate('Gotchas')>>>I = aclass()>>>I.method()Gotchas=1
__slots__ and
super valid to use in your
code?object built-in class (or any other built-in
type). In Python 3.X, all classes are new-style automatically, so this
derivation is not required (but doesn’t hurt); in 2.X, classes with
this explicit derivation are new-style and those without it are
“classic.”type built-in for instances and classes, do
not run generic attribute fetch methods such as __getattr__ for built-in operation methods,
and support a set of advanced extra tools including properties,
descriptors, super, and __slots__ instance attribute lists.self argument (the implied instance), but
static methods do not. Static methods are simple functions nested in
class objects. To make a method static, it must either be run through
a special built-in function or be decorated with decorator syntax.
Python 3.X allows simple functions in a class to be called through the
class without this step, but calls through instances still require
static method declaration.super can mask
later problems when used for single inheritance, and in multiple
inheritance brings with it substantial complexity for an isolated use
case; and both require universal deployment to be most useful.
Evaluating new or advanced tools is a primary task of any engineer,
and is why we explored tradeoffs so carefully in this chapter. This
book’s goal is not to tell you which tools to use, but to underscore
the importance of objectively analyzing them — a task often given too
low a priority in the software field.Adder that exports a method add(self, x, y) that prints a “Not
Implemented” message. Then, define two subclasses of Adder that implement the add method:ListAdderWith anaddmethod that returns the concatenation of its two list arguments
DictAdderWith anaddmethod that returns a new dictionary containing the items in both its two dictionary arguments (any definition of dictionary addition will do)
add
methods.Adder
superclass to save an object in the instance with a constructor (e.g.,
assign self.data a list or a
dictionary), and overload the +
operator with an __add__ method to
automatically dispatch to your add
methods (e.g., X + Y triggers
X.add(X.data,Y)). Where is the best
place to put the constructors and operator overloading methods (i.e.,
in which classes)? What sorts of objects can you add to your class
instances?add methods to accept just one real argument
(e.g., add(self,y)), and add that
one argument to the instance’s current data (e.g., self.data + y). Does this make more sense
than passing two arguments to add?
Would you say this makes your classes more “object-oriented”?MyList that shadows (“wraps”) a
Python list: it should overload most list operators and operations,
including +, indexing, iteration,
slicing, and list methods such as append and sort. See the Python reference manual or
other documentation for a list of all possible methods to support.
Also, provide a constructor for your class that takes an existing list
(or a MyList instance) and copies
its components into an instance attribute. Experiment with your class
interactively. Things to explore:start[:]) to copy the initial value if
it’s a MyList instance?MyList and
a regular list? How about a list and a MyList instance?+ and slicing return? What about
indexing operations?MyList from exercise 2 called MyListSub, which extends MyList to print a message to stdout before each call to the + overloaded operation and counts the number
of such calls. MyListSub should
inherit basic method behavior from MyList. Adding a sequence to a MyListSub should print a message, increment
the counter for + calls, and
perform the superclass’s method. Also, introduce a new method that
prints the operation counters to stdout, and experiment with your class
interactively. Do your counters count calls per instance, or per class
(for all instances of the class)? How would you program the other
option? (Hint: it depends on which object the count members are
assigned to: class members are shared by instances, but self members are per-instance data.)Attrs with methods that intercept
every attribute qualification (both fetches and assignments), and
print messages listing their arguments to stdout. Create an Attrs instance, and experiment with
qualifying it interactively. What happens when you try to use the
instance in expressions? Try adding, indexing, and slicing the
instance of your class. (Note: a fully generic approach based upon
__getattr__ will work in 2.X’s
classic classes but not in 3.X’s new-style classes — which are optional
in 2.X — for reasons noted in Chapter 28, Chapter 31, and Chapter 32, and summarized in the solution to
this exercise.)& and
| operator expressions.for loop. Which methods run
this time?*args
argument form. (Hint: see the function versions of these
algorithms in Chapter 18.) Compute intersections
and unions of multiple operands with your set subclass. How can
you intersect three or more sets, given that & has only two sides?__add__
can catch concatenation, and __getattr__ can pass most named list
method calls like append to the
wrapped list.)__bases__ attribute that returns a
tuple of their superclass objects (the ones listed in parentheses in
the class header). Use __bases__ to
extend the lister.py mix-in
classes we wrote in Chapter 31 so that
they print the names of the immediate superclasses of the instance’s
class. When you’re done, the first line of the string representation
should look like this (your address will almost certainly
vary):<Instance of Sub(Super, Lister), address 7841200:
LunchA container and controller class
CustomerThe actor who buys food
EmployeeThe actor from whom a customer orders
FoodWhat the customer buys
class Lunch:
def __init__(self) # Make/embed Customer and Employee
def order(self, foodName) # Start a Customer order simulation
def result(self) # Ask the Customer what Food it has
class Customer:
def __init__(self) # Initialize my food to None
def placeOrder(self, foodName, employee) # Place order with an Employee
def printFood(self) # Print the name of my food
class Employee:
def takeOrder(self, foodName) # Return a Food, with requested name
class Food:
def __init__(self, name) # Store food nameLunch class’s
constructor should make and embed an instance of Customer and an instance of Employee, and it should export a method
called order. When called, this
order method should ask the
Customer to place an order by
calling its placeOrder method.
The Customer’s placeOrder method should in turn ask the
Employee object for a new
Food object by calling Employee’s takeOrder method.Food objects should store
a food name string (e.g., “burritos”), passed down from Lunch.order, to Customer.placeOrder, to Employee.takeOrder, and finally to
Food’s constructor. The
top-level Lunch class should
also export a method called result, which asks the customer to print
the name of the food it received from the Employee via the order (this can be used
to test your simulation).Lunch needs to pass
either the Employee or itself to
the Customer to allow the Customer to call Employee methods.Lunch class, calling its order method to run an interaction, and then
calling its result method to verify
that the Customer got what he or
she ordered. If you prefer, you can also simply code test cases as
self-test code in the file where your classes are defined, using the
module __name__ trick of Chapter 25. In this simulation, the Customer is the active agent; how would your
classes change if Employee were the
object that initiated customer/employee interaction instead?class
statements to model this taxonomy with Python
inheritance. Then, add a speak method to each of your classes that
prints a unique message, and a reply method in your top-level Animal superclass that simply calls self.speak to invoke the category-specific
message printer in a subclass below (this will kick off an independent
inheritance search from self).
Finally, remove the speak method
from your Hacker class so that it
picks up the default above it. When you’re finished, your classes
should work this way:%python>>>from zoo import Cat, Hacker>>>spot = Cat()>>>spot.reply()# Animal.reply: calls Cat.speakmeow >>>data = Hacker()# Animal.reply: calls Primate.speak>>>data.reply()Hello world!
Scene object to define an action method, and embed instances of the
Customer, Clerk, and Parrot classes (each of which should define
a line method that prints a unique
message). The embedded objects may either inherit from a common
superclass that defines line and
simply provide message text, or define line themselves. In the end, your classes
should operate like this:%python>>>import parrot>>>parrot.Scene().action()# Activate nested objectscustomer: "that's one ex-bird!" clerk: "no it isn't..." parrot: None
1 As a data point, the book Programming Python, a 1,600-page applications programming follow-up to this book that uses 3.X exclusively, neither uses nor needs to accommodate any of the new-style class tools of this chapter, and still manages to build significant programs for GUIs, websites, systems programming, databases, and text. It’s mostly straightforward code that leverages built-in types and libraries to do its work, not obscure and esoteric OOP extensions. When it does use classes, they are relatively simple, providing structure and code factoring. That book’s code is also probably more representative of real-world programming than some in this language tutorial text — which suggests that many of Python’s advanced OOP tools may be artificial, having more to do with language design than practical program goals. Then again, that book has the luxury of restricting its toolset to such code; as soon as your coworker finds a way to use an arcane language feature, all bets are off!
2 As of this chapter’s interaction listings, I’ve started omitting some blank lines and shortening some hex addresses to 32 bits in object displays, to reduce size and clutter. I’m going to assume that by this point in the book, you’ll find such small details irrelevant.
3 Both are opinion pieces in part, but are suggested reading.
The first was eventually retitled “Python’s Super is nifty, but you
can’t use it,” and is today at https://fuhm.net/super-harmful.
Oddly — and despite its subjective tone — the second article (“Python’s
super() considered super!”) alone somehow found its way into
Python’s official library manual; see its link in the manual’s
super section...and consider
demanding that differing opinions be represented more evenly in your
tools’ documentation, or omitted altogether. Python’s manuals are
not the place for personal opinion and one-sided propaganda!
4 This quote is from Monty Python and the Holy Grail (and if you didn’t know that, it may be time to find a copy!).
try/excepttry/finallyraiseassertwith/asPython raises exceptions whenever it detects errors in programs at runtime. You can catch and respond to the errors in your code, or ignore the exceptions that are raised. If an error is ignored, Python’s default exception-handling behavior kicks in: it stops the program and prints an error message. If you don’t want this default behavior, code atrystatement to catch and recover from the exception — Python will jump to yourtryhandler when the error is detected, and your program will resume execution after thetry.
Exceptions can also be used to signal valid conditions without you having to pass result flags around a program or test them explicitly. For instance, a search routine might raise an exception on failure, rather than returning an integer result code — and hoping that the code will never be a valid result!
Sometimes a condition may occur so rarely that it’s hard to justify convoluting your code to handle it in multiple places. You can often eliminate special-case code by handling unusual cases in exception handlers in higher levels of your program. Anassertcan similarly be used to check that conditions are as expected during development.
Finally, because exceptions are a sort of high-level and structured “go to,” you can use them as the basis for implementing exotic control flows. For instance, although the language does not explicitly support backtracking, you can implement it in Python by using exceptions and a bit of support logic to unwind assignments.1 There is no “go to” statement in Python (thankfully!), but exceptions can sometimes serve similar roles; araise, for instance, can be used to jump out of multiple loops.
>>>def fetcher(obj, index):return obj[index]
>>>x = 'spam'>>>fetcher(x, 3)# Like x[3]'m'
>>>fetcher(x, 4)# Default handler - shell interfaceTraceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in fetcher IndexError: string index out of range
>>>fetcher(x, 4)# Default handler - IDLE GUI interfaceTraceback (most recent call last): File "<pyshell#6>", line 1, in <module> fetcher(x, 4) File "<pyshell#3>", line 2, in fetcher return obj[index] IndexError: string index out of range
>>>try:...fetcher(x, 4)...except IndexError:# Catch and recover...print('got exception')... got exception >>>
>>>def catcher():try:fetcher(x, 4)except IndexError:print('got exception')print('continuing')>>>catcher()got exception continuing >>>
>>>try:...raise IndexError# Trigger exception manually...except IndexError:...print('got exception')... got exception
>>> raise IndexError
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError>>> assert False, 'Nobody expects the Spanish Inquisition!'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AssertionError: Nobody expects the Spanish Inquisition!>>>class AlreadyGotOne(Exception): pass# User-defined exception>>>def grail():raise AlreadyGotOne()# Raise an instance>>>try:...grail()...except AlreadyGotOne:# Catch class name...print('got exception')... got exception >>>
>>>class Career(Exception):def __str__(self): return 'So I became a waiter...'>>>raise Career()Traceback (most recent call last): File "<stdin>", line 1, in <module> __main__.Career: So I became a waiter... >>>
>>>try:...fetcher(x, 3)...finally:# Termination actions...print('after fetch')... 'm' after fetch >>>
fetcher(x, 3)
print('after fetch')>>>def after():try:fetcher(x, 4)finally:print('after fetch')print('after try?')>>>after()after fetch Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in after File "<stdin>", line 2, in fetcher IndexError: string index out of range >>>
>>>def after():try:fetcher(x, 3)finally:print('after fetch')print('after try?')>>>after()after fetch after try? >>>
>>>with open('lumberjack.txt', 'w') as file:# Always close file on exitfile.write('The larch!\n')
try/except statements to catch and recover from
exceptions that are raised within its nested code block. Once an
exception is caught, the exception is terminated and your program
continues after the try.raise and assert statements can be used to trigger an
exception, exactly as if it had been raised by Python itself. In
principle, you can also raise an exception by making a programming
mistake, but that’s not usually an explicit goal!try/finally statement can be used to ensure
actions are run after a block of code exits, regardless of whether the
block raises an exception or not. The with/as
statement can also be used to ensure termination actions are run, but
only when processing object types that support it.1 But true backtracking is not part of the Python
language. Backtracking undoes all computations before it
jumps, but Python exceptions do not: variables assigned
between the time a try
statement is entered and the time an exception is raised are
not reset to their prior values. Even the generator functions
and expressions we met in Chapter 20 don’t do full
backtracking — they simply respond to next(G) requests by restoring state
and resuming. For more on backtracking, see books on
artificial intelligence or the Prolog or Icon programming
languages.
try:
statements # Run this main action first
except name1:
statements # Run if name1 is raised during try block
except (name2, name3):
statements # Run if any of these exceptions occur
except name4 as var:
statements # Run if name4 is raised, assign instance raised to var
except:
statements # Run for all other exceptions raised
else:
statements # Run if no exception was raised during try blocktry block’s statements are running, and
the exception matches one that the statement
names, Python jumps back to the try and runs the statements under the
first except clause that matches
the raised exception, after assigning the raised exception object to
the variable named after the as
keyword in the clause (if present). After the except block runs, control then resumes
below the entire try statement
(unless the except block itself
raises another exception, in which case the process is started anew
from this point in the code).try block’s statements are running, but
the exception does not match one that the
statement names, the exception is propagated up to the next most
recently entered try statement
that matches the exception; if no such matching try statement can be found and the search
reaches the top level of the process, Python kills the program and
prints a default error message.try block’s statements are
running, Python runs the statements under the else line (if present), and control then
resumes below the entire try
statement.| Clause form | Interpretation |
|---|---|
except clauses that list
no exception name (except:)
catch all exceptions not previously listed in
the try statement.except clauses that list
a set of exceptions in parentheses (except (e1, e2, e3):) catch
any of the listed exceptions.try:
action()
except NameError:
...
except IndexError:
...
except KeyError:
...
except (AttributeError, TypeError, SyntaxError):
...
else:
...try:
action()
except NameError:
... # Handle NameError
except IndexError:
... # Handle IndexError
except:
... # Handle all other exceptions
else:
... # Handle the no-exception casetry:
action()
except:
... # Catch all possible exceptionstry:
action()
except Exception:
... # Catch all possible exceptions, except exitstry:
...run code...
except IndexError:
...handle exception...
# Did we get here because the try failed or not?try:
...run code...
except IndexError:
...handle exception...
else:
...no exception occurred...try:
...run code...
...no exception occurred...
except IndexError:
...handle exception...def gobad(x, y):
return x / y
def gosouth(x):
print(gobad(x, 0))
gosouth(1)% python bad.py
Traceback (most recent call last):
File "bad.py", line 7, in <module>
gosouth(1)
File "bad.py", line 5, in gosouth
print(gobad(x, 0))
File "bad.py", line 2, in gobad
return x / y
ZeroDivisionError: division by zerodef kaboom(x, y):
print(x + y) # Trigger TypeError
try:
kaboom([0, 1, 2], 'spam')
except TypeError: # Catch and recover here
print('Hello world!')
print('resuming here') # Continue here if exception or not% python kaboom.py
Hello world!
resuming heretry:
statements # Run this action first
finally:
statements # Always run this code on the way outtry block is running, Python
continues on to run the finally
block, and then continues execution past the try statement.try block’s run, Python still comes
back and runs the finally block,
but it then propagates the exception up to a previously entered
try or the top-level default
handler; the program does not resume execution below the finally clause’s try statement. That is, the finally block is run even if an exception is
raised, but unlike an except, the
finally does not terminate the
exception — it continues being raised after the finally block runs.class MyError(Exception): pass
def stuff(file):
raise MyError()
file = open('data', 'w') # Open an output file (this can fail too)
try:
stuff(file) # Raises exception
finally:
file.close() # Always close file to flush output buffers
print('not reached') # Continue here only if no exceptiontry:# Merged formmain-actionexcept Exception1:handler1except Exception2:# Catch exceptionshandler2... else:# No-exception handlerelse-blockfinally:# The finally encloses all elsefinally-block
try -> except -> else -> finally
try:# Format 1statementsexcept [type[asvalue]]:# [type [, value]] in Python 2.Xstatements[except [type[asvalue]]:statements]* [else:statements] [finally:statements] try:# Format 2statementsfinally:statements
try:# Nested equivalent to merged formtry:main-actionexcept Exception1:handler1except Exception2:handler2... else:no-errorfinally:cleanup
# File mergedexc.py (Python 3.X + 2.X)
sep = '-' * 45 + '\n'
print(sep + 'EXCEPTION RAISED AND CAUGHT')
try:
x = 'spam'[99]
except IndexError:
print('except run')
finally:
print('finally run')
print('after run')
print(sep + 'NO EXCEPTION RAISED')
try:
x = 'spam'[3]
except IndexError:
print('except run')
finally:
print('finally run')
print('after run')
print(sep + 'NO EXCEPTION RAISED, WITH ELSE')
try:
x = 'spam'[3]
except IndexError:
print('except run')
else:
print('else run')
finally:
print('finally run')
print('after run')
print(sep + 'EXCEPTION RAISED BUT NOT CAUGHT')
try:
x = 1 / 0
except IndexError:
print('except run')
finally:
print('finally run')
print('after run')c:\code> py −3 mergedexc.py
---------------------------------------------
EXCEPTION RAISED AND CAUGHT
except run
finally run
after run
---------------------------------------------
NO EXCEPTION RAISED
finally run
after run
---------------------------------------------
NO EXCEPTION RAISED, WITH ELSE
else run
finally run
after run
---------------------------------------------
EXCEPTION RAISED BUT NOT CAUGHT
finally run
Traceback (most recent call last):
File "mergedexc.py", line 39, in <module>
x = 1 / 0
ZeroDivisionError: division by zeroraiseinstance# Raise instance of classraiseclass# Make and raise instance of class: makes an instanceraise# Reraise the most recent exception
raise IndexError# Class (instance created)raise IndexError()# Instance (created in statement)
exc = IndexError() # Create instance ahead of time
raise exc
excs = [IndexError, TypeError]
raise excs[0]try:
...
except IndexError as X: # X assigned the raised instance object
...class MyExc(Exception): pass
...
raise MyExc('spam') # Exception class with constructor args
...
try:
...
except MyExc as X: # Instance attributes available in handler
print(X.args)c:\code>py −2>>>try:...1 / 0...except Exception as X:# 2.X does not localize X either way...print X... integer division or modulo by zero >>>XZeroDivisionError('integer division or modulo by zero',)
>>>try:...1 / 0...except Exception, X:...print X... integer division or modulo by zero >>>XZeroDivisionError('integer division or modulo by zero',)
c:\code>py −3>>>try:...1 / 0...except Exception, X:SyntaxError: invalid syntax >>>try:...1 / 0...except Exception as X:# 3.X localizes 'as' names to except block...print(X)... division by zero >>>XNameError: name 'X' is not defined
>>>X = 99>>>try:...1 / 0...except Exception as X:# 3.X localizes _and_ removes on exit!...print(X)... division by zero >>>XNameError: name 'X' is not defined >>>X = 99>>>{X for X in 'spam'}# 2.X/3.X localizes only: not removed{'s', 'a', 'p', 'm'} >>>X99
>>>try:...1 / 0...except Exception as X:# Python removes this reference...print(X)...Saveit = X# Assign exc to retain exc if needed... division by zero >>>XNameError: name 'X' is not defined >>>SaveitZeroDivisionError('division by zero',)
>>>try:...raise IndexError('spam')# Exceptions remember arguments...except IndexError:...print('propagating')...raise# Reraise most recent exception... propagating Traceback (most recent call last): File "<stdin>", line 2, in <module> IndexError: spam
raisenewexceptionfromotherexception
>>>try:...1 / 0...except Exception as E:...raise TypeError('Bad') from E# Explicitly chained exceptions... Traceback (most recent call last): File "<stdin>", line 2, in <module> ZeroDivisionError: division by zero The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 4, in <module> TypeError: Bad
>>>try:...1 / 0...except:...badname# Implicitly chained exceptions... Traceback (most recent call last): File "<stdin>", line 2, in <module> ZeroDivisionError: division by zero During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 4, in <module> NameError: name 'badname' is not defined
>>>try:...try:...raise IndexError()...except Exception as E:...raise TypeError() from E...except Exception as E:...raise SyntaxError() from E... Traceback (most recent call last): File "<stdin>", line 3, in <module> IndexError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 5, in <module> TypeError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 7, in <module> SyntaxError: None
try:
try:
1 / 0
except:
badname
except:
open('nonesuch')raise newexception from None
asserttest,data# Thedatapart is optional
if __debug__:
if not test:
raise AssertionError(data)def f(x):
assert x < 0, 'x must be negative'
return x ** 2
% python
>>> import asserter
>>> asserter.f(1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".\asserter.py", line 2, in f
assert x < 0, 'x must be negative'
AssertionError: x must be negativedef reciprocal(x):
assert x != 0 # A generally useless assert!
return 1 / x # Python checks for zero automaticallyfrom __future__ import with_statement
withexpression[asvariable]:with-block
with open(r'C:\misc\data') as myfile:
for line in myfile:
print(line)
...more code here...myfile = open(r'C:\misc\data')
try:
for line in myfile:
print(line)
...more code here...
finally:
myfile.close()lock = threading.Lock()# After: import threadingwith lock: # critical section of code...access shared resources...
with decimal.localcontext() as ctx: # After: import decimal
ctx.prec = 2
x = decimal.Decimal('1.00') / decimal.Decimal('3.00')__enter__ and __exit__ methods.__enter__ method is called. The value it returns is assigned to the
variable in the as clause if
present, or simply discarded otherwise.with
block is executed.with block raises an
exception, the __exit__(type, value,
traceback) method is called with the exception
details. These are the same three values returned by sys.exc_info, described in the Python
manuals and later in this part of the book. If this method returns a
false value, the exception is reraised; otherwise, the exception is
terminated. The exception should normally be reraised so that it is
propagated outside the with
statement.with block does not
raise an exception, the __exit__
method is still called, but its type,
value, and
traceback arguments are all passed in as
None.class TraceBlock:
def message(self, arg):
print('running ' + arg)
def __enter__(self):
print('starting with block')
return self
def __exit__(self, exc_type, exc_value, exc_tb):
if exc_type is None:
print('exited normally\n')
else:
print('raise an exception! ' + str(exc_type))
return False # Propagate
if __name__ == '__main__':
with TraceBlock() as action:
action.message('test 1')
print('reached')
with TraceBlock() as action:
action.message('test 2')
raise TypeError
print('not reached')c:\code> py −3 withas.py
starting with block
running test 1
reached
exited normally
starting with block
running test 2
raise an exception! <class 'TypeError'>
Traceback (most recent call last):
File "withas.py", line 22, in <module>
raise TypeError
TypeErrorwith open('data') as fin, open('res', 'w') as fout:
for line in fin:
if 'some key' in line:
fout.write(line)with A() as a, B() as b:
...statements...with A() as a:
with B() as b:
...statements...>>>with open('script1.py') as f1, open('script2.py') as f2:...for pair in zip(f1, f2):...print(pair)... ('# A first Python script\n', 'import sys\n') ('import sys # Load a library module\n', 'print(sys.path)\n') ('print(sys.platform)\n', 'x = 2\n') ('print(2 ** 32) # Raise 2 to a power\n', 'print(x ** 32)\n')
with open('script1.py') as f1, open('script2.py') as f2:
for (linenum, (line1, line2)) in enumerate(zip(f1, f2)):
if line1 != line2:
print('%s\n%r\n%r' % (linenum, line1, line2))for pair in zip(open('script1.py'), open('script2.py')): # Same effect, auto close
print(pair)>>>with open('script2.py') as fin, open('upper.py', 'w') as fout:...for line in fin:...fout.write(line.upper())... >>>print(open('upper.py').read())IMPORT SYS PRINT(SYS.PATH) X = 2 PRINT(X ** 32)
fin = open('script2.py')
fout = open('upper.py', 'w')
for line in fin: # Same effect as preceding code, auto close
fout.write(line.upper())fin = open('script2.py')
fout = open('upper.py', 'w')
try: # Same effect but explicit close on error
for line in fin:
fout.write(line.upper())
finally:
fin.close()
fout.close()try statement catches and
recovers from exceptions — it specifies a block of code to run, and one
or more handlers for exceptions that may be raised during the block’s
execution.try statement are try/except/else (for catching exceptions) and try/finally (for specifying cleanup actions that
must occur whether an exception is raised or not). Through Python 2.4,
these were separate statements that could be combined by syntactic
nesting; in 2.5 and later, except
and finally blocks may be mixed in
the same statement, so the two statement forms are merged. In the
merged form, the finally is still
run on the way out of the try,
regardless of what exceptions may have been raised or handled. In
fact, the merged form is equivalent to nesting a try/except/else in a try/finally, and the two still have logically
distinct roles.raise statement raises
(triggers) an exception. Python raises built-in exceptions on errors
internally, but your scripts can trigger built-in or user-defined
exceptions with raise, too.assert statement raises
an AssertionError exception if a
condition is false. It works like a conditional raise statement wrapped up in an if statement, and can be disabled with a
–O switch.with/as statement is designed to automate startup
and termination activities that must occur around a block of code. It
is roughly like a try/finally statement in that its exit actions
run whether an exception occurred or not, but it allows a richer
object-based protocol for specifying entry and
exit actions, and may reduce code size. Still, it’s not quite as
general, as it applies only to objects that support its protocol; try
handles many more use cases.1 As mentioned in the prior chapter, the text of error messages and stack traces tends to vary slightly over time and shells. Don’t be alarmed if your error messages don’t exactly match mine. When I ran this example in Python 3.3’s IDLE GUI, for instance, its error message text showed filenames with full absolute directory paths.
2 Unless Python crashes completely, of course. It does a good job of avoiding this, though, by checking all possible errors as a program runs. When a program does crash hard, it is usually due to a bug in linked-in C extension code, outside of Python’s scope.
try statements.try handler — instances have access to both
attached state information and callable methods.C:\code>C:\Python25\python>>>myexc = "My exception string"# Were we ever this young?...>>>try:...raise myexc...except myexc:...print('caught')... caught
C:\code>py −3>>>raise 'spam'TypeError: exceptions must derive from BaseException C:\code>py −2>>>raise 'spam'TypeError: exceptions must be old-style classes or derived from BaseException, ...etc
except clauses by Python’s is test.except clause if that except clause names the exception
instance’s class or any superclass of it.class General(Exception): pass
class Specific1(General): pass
class Specific2(General): pass
def raiser0():
X = General() # Raise superclass instance
raise X
def raiser1():
X = Specific1() # Raise subclass instance
raise X
def raiser2():
X = Specific2() # Raise different subclass instance
raise X
for func in (raiser0, raiser1, raiser2):
try:
func()
except General: # Match General or any subclass of it
import sys
print('caught: %s' % sys.exc_info()[0])
C:\code> python classexc.py
caught: <class '__main__.General'>
caught: <class '__main__.Specific1'>
caught: <class '__main__.Specific2'>Classes used to build exception category trees have very few requirements — in fact, in this example they are mostly empty, with bodies that do nothing butpass. Notice, though, how the top-level class here inherits from the built-inExceptionclass. This is required in Python 3.X; Python 2.X allows standalone classic classes to serve as exceptions too, but it requires new-style classes to be derived from built-in exception classes just as in 3.X. Although we don’t employ it here, becauseExceptionprovides some useful behavior we’ll meet later, it’s a good idea to inherit from it in either Python.
In this code, we call classes to make instances for theraisestatements. In the class exception model, we always raise and catch a class instance object. If we list a class name without parentheses in araise, Python calls the class with no constructor argument to make an instance for us. Exception instances can be created before theraise, as done here, or within theraisestatement itself.
This code includes functions that raise instances of all three of our classes as exceptions, as well as a top-leveltrythat calls the functions and catchesGeneralexceptions. The sametryalso catches the two specific exceptions, because they are subclasses ofGeneral— members of its category.
The exception handler here uses thesys.exc_infocall — as we’ll see in more detail in the next chapter, it’s how we can grab hold of the most recently raised exception in a generic fashion. Briefly, the first item in its result is the class of the exception raised, and the second is the actual instance raised. In a generalexceptclause like the one here that catches all classes in a category,sys.exc_infois one way to determine exactly what’s occurred. In this particular case, it’s equivalent to fetching the instance’s__class__attribute. As we’ll see in the next chapter, thesys.exc_infoscheme is also commonly used with emptyexceptclauses that catch everything.
class General(Exception): pass
class Specific1(General): pass
class Specific2(General): pass
def raiser0(): raise General()
def raiser1(): raise Specific1()
def raiser2(): raise Specific2()
for func in (raiser0, raiser1, raiser2):
try:
func()
except General as X: # X is the raised instance
print('caught: %s' % X.__class__) # Same as sys.exc_info()[0]try:
func()
except (General, Specific1, Specific2): # Catch any of these
...# mathlib.pyclass Divzero(Exception): pass class Oflow(Exception): pass def func(): ... raise Divzero()...and so on...
# client.pyimport mathlib try: mathlib.func(...) except (mathlib.Divzero, mathlib.Oflow):...handle and recover...
# mathlib.py
class Divzero(Exception): pass
class Oflow(Exception): pass
class Uflow(Exception): pass# client.pytry: mathlib.func(...) except (mathlib.Divzero, mathlib.Oflow,mathlib.Uflow):...handle and recover...
# client.pytry: mathlib.func(...) except:# Catch everything here (or catch Exception super)...handle and recover...
# mathlib.pyclass NumErr(Exception): pass class Divzero(NumErr): pass class Oflow(NumErr): pass def func(): ... raise DivZero()...and so on...
# client.pyimport mathlib try: mathlib.func(...) except mathlib.NumErr:...report and recover...
# mathlib.py
...
class Uflow(NumErr): passBaseException: topmost root,
printing and constructor defaultsThe top-level root superclass of exceptions. This class is not supposed to be directly inherited by user-defined classes (useExceptioninstead). It provides default printing and state retention behavior inherited by subclasses. If thestrbuilt-in is called on an instance of this class (e.g., byargsattribute as a tuple.
Exception: root of
user-defined exceptionsThe top-level root superclass of application-related exceptions. This is an immediate subclass ofBaseExceptionand is a superclass to every other built-in exception, except the system exit event classes (SystemExit,KeyboardInterrupt, andGeneratorExit). Nearly all user-defined classes should inherit from this class, notBaseException. When this convention is followed, namingExceptionin atrystatement’s handler ensures that your program will catch everything but system exit events, which should normally be allowed to pass. In effect,Exceptionbecomes a catchall intrystatements and is more accurate than an emptyexcept.
ArithmeticError: root of
numeric errorsLookupError: root of indexing
errors>>>import exceptions>>>help(exceptions)...lots of text omitted...
ArithmeticError
in a try, you will catch
any kind of numeric error raised.ZeroDivisionError, you will intercept just that specific type
of error, and no others.try:
action()
except Exception: # Exits not caught here
...handle all application exceptions...
else:
...handle no-exception case...c:\temp>py −3.2>>>try:...f = open('nonesuch.txt')...except IOError as V:...if V.errno == 2:# Or errno.N, V.args[0]...print('No such file')...else:...raise# Propagate others... No such file
c:\temp>py −3.3>>>try:...f = open('nonesuch.txt')...except FileNotFoundError:...print('No such file')... No such file
>>>raise IndexError# Same as IndexError(): no argumentsTraceback (most recent call last): File "<stdin>", line 1, in <module> IndexError >>>raise IndexError('spam')# Constructor argument attached, printedTraceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: spam >>>I = IndexError('spam')# Available in object attribute>>>I.args('spam',) >>>print(I)# Displays args when printed manuallyspam
>>>class E(Exception): pass... >>>raise ETraceback (most recent call last): File "<stdin>", line 1, in <module> __main__.E >>>raise E('spam')Traceback (most recent call last): File "<stdin>", line 1, in <module> __main__.E: spam >>>I = E('spam')>>>I.args('spam',) >>>print(I)spam
>>>try:...raise E('spam')...except E as X:...print(X)# Displays and saves constructor arguments...print(X.args)...print(repr(X))... spam ('spam',) E('spam',) >>>try:# Multiple arguments save/display a tuple...raise E('spam', 'eggs', 'ham')...except E as X:...print('%s %s' % (X, X.args))... ('spam', 'eggs', 'ham') ('spam', 'eggs', 'ham')
>>>class MyBad(Exception): pass... >>>try:...raise MyBad('Sorry--my mistake!')...except MyBad as X:...print(X)... Sorry--my mistake!
>>> raise MyBad('Sorry--my mistake!')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
__main__.MyBad: Sorry--my mistake!>>>class MyBad(Exception):...def __str__(self):...return 'Always look on the bright side of life...'... >>>try:...raise MyBad()...except MyBad as X:...print(X)... Always look on the bright side of life... >>>raise MyBad()Traceback (most recent call last): File "<stdin>", line 1, in <module> __main__.MyBad: Always look on the bright side of life...
>>>class E(Exception):def __repr__(self): return 'Not called!'>>>raise E('spam')... __main__.E: spam >>>class E(Exception):def __str__(self): return 'Called!'>>>raise E('spam')... __main__.E: Called!
>>>class FormatError(Exception):def __init__(self, line, file):self.line = lineself.file = file>>>def parser():raise FormatError(42, file='spam.txt')# When error found>>>try:...parser()...except FormatError as X:...print('Error at: %s %s' % (X.file, X.line))... Error at: spam.txt 42
>>>class FormatError(Exception): pass# Inherited constructor>>>def parser():raise FormatError(42, 'spam.txt')# No keywords allowed!>>>try:...parser()...except FormatError as X:...print('Error at:', X.args[0], X.args[1])# Not specific to this app... Error at: 42 spam.txt
from __future__ import print_function # 2.X compatibility
class FormatError(Exception):
logfile = 'formaterror.txt'
def __init__(self, line, file):
self.line = line
self.file = file
def logerror(self):
log = open(self.logfile, 'a')
print('Error at:', self.file, self.line, file=log)
def parser():
raise FormatError(40, 'spam.txt')
if __name__ == '__main__':
try:
parser()
except FormatError as exc:
exc.logerror()c:\code>del formaterror.txtc:\code>py −3 excparse.pyc:\code>py −2 excparse.pyc:\code>type formaterror.txtError at: spam.txt 40 Error at: spam.txt 40
class CustomFormatError(FormatError):
def logerror(self):
...something unique here...
raise CustomFormatError(...)BaseException; most programs inherit from
its Exception subclass, to support
catchall handlers for normal kinds of exceptions.args). In exception handlers, you list a
variable to be assigned to the raised instance, then go through this
name to access attached state information and call any methods defined
in the class.__str__
operator overloading method. For simpler needs, built-in exception
superclasses automatically display anything you pass to the class
constructor. Operations like print
and str automatically fetch the
display string of an exception object when it is printed either
explicitly or as part of an error message.1 As a clever student of mine suggested, the library module could
also provide a tuple object that contains all the exceptions the
library can possibly raise — the client could then import the tuple and
name it in an except clause to
catch all the library’s exceptions (recall that including a tuple in
an except means catch
any of its exceptions). When new exceptions are
added later, the library can just expand the exported tuple. This
would work, but you’d still need to keep the tuple up-to-date with
raised exceptions inside the library module. Also, class hierarchies
offer more benefits than just categories — they also support inherited
state and methods and a customization model that individual exceptions
do not.
def action2():
print(1 + []) # Generate TypeError
def action1():
try:
action2()
except TypeError: # Most recent matching try
print('inner try')
try:
action1()
except TypeError: # Here, only if action1 re-raises
print('outer try')
% python nestexc.py
inner trytry:
try:
action2()
except TypeError: # Most recent matching try
print('inner try')
except TypeError: # Here, only if nested handler re-raises
print('outer try')>>>try:...try:...raise IndexError...finally:...print('spam')...finally:...print('SPAM')... spam SPAM Traceback (most recent call last): File "<stdin>", line 3, in <module> IndexError
def raise1(): raise IndexError
def noraise(): return
def raise2(): raise SyntaxError
for func in (raise1, noraise, raise2):
print('<%s>' % func.__name__)
try:
try:
func()
except IndexError:
print('caught IndexError')
finally:
print('finally run')
print('...')% python except-finally.py
<raise1>
caught IndexError
finally run
...
<noraise>
finally run
...
<raise2>
finally run
Traceback (most recent call last):
File "except-finally.py", line 9, in <module>
func()
File "except-finally.py", line 3, in raise2
def raise2(): raise SyntaxError
SyntaxError: None>>>class Exitloop(Exception): pass... >>>try:...while True:...while True:...for i in range(10):...if i > 3: raise Exitloop# break exits just one level...print('loop3: %s' % i)...print('loop2')...print('loop1')...except Exitloop:...print('continuing')# Or just pass, to move on... loop3: 0 loop3: 1 loop3: 2 loop3: 3 continuing >>>i4
while True:
try:
line = input() # Read line from stdin (raw_input in 2.X)
except EOFError:
break # Exit loop at end-of-file
else:
...process next line here...class Found(Exception): pass
def searcher():
if ...success...:
raise Found() # Raise exceptions instead of returning flags
else:
return
try:
searcher()
except Found: # Exception if item was found
...success...
else: # else returned: not found
...failure...class Failure(Exception): pass
def searcher():
if ...success...:
return ...founditem...
else:
raise Failure()
try:
item = searcher()
except Failure:
...not found...
else:
...use item here...myfile = open(r'C:\code\textdata', 'w')
try:
...process myfile...
finally:
myfile.close()with open(r'C:\code\textdata', 'w') as myfile:
...process myfile...myfile = open(filename, 'w')# Traditional form...process myfile...myfile.close() with open(filename) as myfile:# Context manager form...process myfile...
try:
...run program...
except: # All uncaught exceptions come here
import sys
print('uncaught!', sys.exc_info()[0], sys.exc_info()[1])import sys
log = open('testlog', 'a')
from testapi import moreTests, runNextTest, testName
def testdriver():
while moreTests():
try:
runNextTest()
except:
print('FAILED', testName(), sys.exc_info()[:2], file=log)
else:
print('PASSED', testName(), file=log)
testdriver()try:
...
except:
# sys.exc_info()[0:2] are the exception class and instancetype is the exception class of the
exception being handled.value is the exception class
instance that was raised.traceback is a traceback
object that represents the call stack at the point where the
exception originally occurred, and used by the traceback module to generate error
messages.try:
...
except General as instance:
# instance.__class__ is the exception classtry:
...
except General as instance:
# instance.method() does the right thing for this instanceimport traceback
def inverse(x):
return 1 / x
try:
inverse(0)
except Exception:
traceback.print_exc(file=open('badly.exc', 'w'))
print('Bye')c:\code>python badly.pyBye c:\code>type badly.excTraceback (most recent call last): File "badly.py", line 7, in <module> inverse(0) File "badly.py", line 4, in inverse return 1 / x ZeroDivisionError: division by zero
try statements. For example,
operations that interface with system state (file opens, socket
calls, and the like) are prime candidates for try.try/finally statements to guarantee their
execution, unless a context manager is available as a with/as
option. The try/finally statement form allows you to run
code whether exceptions occur or not in arbitrary scenarios.try
statement, rather than littering the function itself with many
try statements. That way, all
exceptions in the function percolate up to the try around the call, and you reduce the
amount of code within the function.def func():
try:
... # IndexError is raised in here
except:
... # But everything comes here and dies!
try:
func()
except IndexError: # Exception should be processed here
...import sys
def bye():
sys.exit(40) # Crucial error: abort now!
try:
bye()
except:
print('got it') # Oops--we ignored the exit
print('continuing...')
% python exiter.py
got it
continuing...try:
bye()
except Exception: # Won't catch exits, but _will_ catch many others
...mydictionary = {...}
...
try:
x = myditctionary['spam'] # Oops: misspelled
except:
x = None # Assume we got KeyError
...continue here with x...try:
...
except (MyExcept1, MyExcept2): # Breaks if you add a MyExcept3 later
... # Nonerrors
else:
... # Assumed to be an errortry:
...
except SuccessCategoryName: # OK if you add a MyExcept3 subclass later
... # Nonerrors
else:
... # Assumed to be an errorBuilt-in types like strings, lists, and dictionaries make it easy to write simple programs fast.
For more demanding tasks, you can extend Python by writing your own functions, modules, and classes.
Although we don’t cover this topic in this book, Python can also be extended with modules written in an external language like C or C++.
PyDoc’shelpfunction and HTML interfaces were introduced in Chapter 15. PyDoc provides a documentation system for your modules and objects, integrates with Python’s docstrings syntax, and is a standard part of the Python system. See Chapter 15 and Chapter 4 for more documentation source hints.
Because Python is such a dynamic language, some programming errors are not reported until your program runs (even syntax errors are not caught until a file is run or imported). This isn’t a big drawback — as with most languages, it just means that you have to test your Python code before shipping it. At worst, with Python you essentially trade a compile phase for an initial testing phase. Furthermore, Python’s dynamic nature, automatic error messages, and exception model make it easier and quicker to find and fix errors than it is in some other languages. Unlike C, for example, Python does not crash completely on errors.Still, tools can help here too. The PyChecker and PyLint systems provide support for catching common errors ahead of time, before your script runs. They serve similar roles to the lint program in C development. Some Python developers run their code through PyChecker prior to testing or delivery, to catch any lurking potential problems. In fact, it’s not a bad idea to try this when you’re first starting out — some of these tools’ warnings may help you learn to spot and avoid common Python mistakes. PyChecker and PyLint are third-party open source packages, available at the PyPI website or your friendly neighborhood web search engine. They may appear in IDE GUIs as well.
unittest)In Chapter 25, we learned how to add self-test code to a Python file by using the__name__ == '__main__'trick at the bottom of the file — a simple unit-testing protocol. For more advanced testing purposes, Python comes with two testing support tools. The first, PyUnit (calledunittestin the library manual), provides an object-oriented class framework for specifying and customizing test cases and expected results. It mimics the JUnit framework for Java. This is a sophisticated class-based unit testing system; see the Python library manual for details.
doctestThedocteststandard library module provides a second and simpler approach to regression testing, based upon Python’s docstrings feature. Roughly, to usedoctest, you cut and paste a log of an interactive testing session into the docstrings of your source files.doctestthen extracts your docstrings, parses out the test cases and results, and reruns the tests to verify the expected results.doctest’s operation can be tailored in a variety of ways; see the library manual for more details.
We discussed IDEs for Python in Chapter 3. IDEs such as IDLE provide a graphical environment for editing, running, debugging, and browsing your Python programs. Some advanced IDEs — such as Eclipse, Komodo, NetBeans, and others listed in Chapter 3 — may support additional development tasks, including source control integration, code refactoring, project management tools, and more. See Chapter 3, the text editors page at http://www.python.org, and your favorite web search engine for more on available IDEs and GUI builders for Python.
Because Python is so high-level and dynamic, intuitions about performance gleaned from experience with other languages usually don’t apply to Python code. To truly isolate performance bottlenecks in your code, you need to add timing logic with clock tools in thetimeortimeitmodules, or run your code under theprofilemodule. We saw an example of the timing modules at work when comparing the speed of iteration tools and Pythons in Chapter 21.Profiling is usually your first optimization step — code for clarity, then profile to isolate bottlenecks, and then time alternative codings of the slow parts of your program. For the second of these steps,profileis a standard library module that implements a source code profiler for Python. It runs a string of code you provide (e.g., a script file import, or a call to a function) and then, by default, prints a report to the standard output stream that gives performance statistics — number of calls to each function, time spent in each function, and more.Theprofilemodule can be run as a script or imported, and it may be customized in various ways; for example, it can save run statistics to a file to be analyzed later with thepstatsmodule. To profile interactively, import theprofilemodule and callprofile.run('code'), passing in the code you wish to profile as a string (e.g., a call to a function, an import of a file, or code read from a file). To profile from a system shell command line, use a command of the formpython -m profile main.pyargs(see Appendix A for more on this format). Also see Python’s standard library manuals for other profiling options; thecProfilemodule, for example, has identical interfaces toprofilebut runs with less overhead, so it may be better suited to profiling long-running programs.
We also discussed debugging options in Chapter 3 (see its sidebar “Debugging Python Code”). As a review, most development IDEs for Python support GUI-based debugging, and the Python standard library also includes a source code debugger module calledpdb. This module provides a command-line interface and works much like common C language debuggers (e.g., dbx, gdb).Much like the profiler, the pdb debugger can be run either interactively or from a command line and can be imported and called from a Python program. To use it interactively, import the module, start running code by calling apdbfunction (e.g.,pdb.run('main()')), and then type debugging commands from pdb’s interactive prompt. To launch pdb from a system shell command line, use a command of the formpython -m pdb main.pyargs. pdb also includes a useful postmortem analysis call,pdb.pm(), which starts the debugger after an exception has been encountered, possibly in conjunction with Python’s-iflag. See Appendix A for more on these tools.Because IDEs such as IDLE also include point-and-click debugging interfaces, pdb isn’t as critical a tool today, except when a GUI isn’t available or when more control is desired. See Chapter 3 for tips on using IDLE’s debugging GUI interfaces. Really, neither pdb nor IDEs seem to be used much in practice — as noted in Chapter 3, most programmers either insert
In Chapter 2, we introduced common tools for packaging Python programs. py2exe, PyInstaller, and others listed in that chapter can package byte code and the Python Virtual Machine into “frozen binary” standalone executables, which don’t require that Python be installed on the target machine and hide your system’s code. In addition, we learned in Chapter 2 that Python programs may be shipped in their source (.py) or byte code (.pyc) forms, and that import hooks support special packaging techniques such as automatic extraction of .zip files and byte code encryption.We also briefly met the standard library’sdistutilsmodules, which provide packaging options for Python modules and packages, and C-coded extensions; see the Python manuals for more details. The emerging Python “eggs” third-party packaging system provides another alternative that also accounts for dependencies; search the Web for more details.
When speed counts, there are a handful of options for optimizing your programs. The PyPy system described in Chapter 2 provides a just-in-time compiler for translating Python byte code to binary machine code, and Shed Skin offers a Python-to-C++ translator. You may also occasionally see .pyo optimized byte code files, generated and run with the-OPython command-line flag discussed in Chapter 22 and Chapter 34, and to be deployed in Chapter 39; because this provides a very modest performance boost, however, it is not commonly used except to remove debugging code.As a last resort, you can also move parts of your program to a compiled language such as C to boost performance. See the book Programming Python and the Python standard manuals for more on C extensions. In general, Python’s speed tends to also improve over time, so upgrading to later releases may improve speed too — once you verify that they are faster for your code, that is (though largely repaired since, Python 3.0’s initial release was up to 1000X slower than 2.X on some IO operations!).
We’ve met a variety of core language features in this text that will also tend to become more useful once you start coding larger projects. These include module packages (Chapter 24), class-based exceptions (Chapter 34), class pseudoprivate attributes (Chapter 31), documentation strings (Chapter 15), module path configuration files (Chapter 22), hiding names fromfrom *with__all__lists and_X-style names (Chapter 25), adding self-test code with the__name__ == '__main__'trick (Chapter 25), using common design rules for functions and modules (Chapter 17, Chapter 19, and Chapter 25), using object-oriented design patterns (Chapter 31 and others), and so on.
try/except. Write a function called oops that explicitly raises an IndexError exception when called. Then write
another function that calls oops
inside a try/except statement to catch the error. What
happens if you change oops to raise
a KeyError
instead of an IndexError? Where do
the names KeyError and IndexError come from? (Hint: recall that all
unqualified names generally come from one of four scopes.)oops function you just wrote to
raise an exception you define yourself, called MyError. Identify your exception with a
class (unless you’re using Python 2.5 or earlier, you must). Then,
extend the try statement in the
catcher function to catch this exception and its instance in addition
to IndexError, and print the
instance you catch.safe(func, *pargs, **kargs) that
runs any function with any number of positional and/or keyword
arguments by using the * arbitrary
arguments header and call syntax, catches any exception raised while
the function runs, and prints the exception using the exc_info call in the sys module. Then use your safe function to run your oops function from exercise 1 or 2. Put
safe in a module file called
exctools.py, and pass it the
oops function interactively. What
kind of error messages do you get? Finally, expand safe to also print a Python stack trace when
an error occurs by calling the built-in print_exc function in the standard traceback module; see earlier in this
chapter, and consult the Python library reference manual for usage
details. We could probably code safe as a function
decorator using Chapter 32
techniques, but we’ll have to move on to the next part of the book to
learn fully how (see the solutions for a preview).1 A related call, os._exit,
also ends a program, but via an immediate termination — it skips
cleanup actions, including any registered with the atexit module noted earlier, and cannot be
intercepted with try/except or try/finally blocks. It is usually used only in
spawned child processes, a topic beyond this book’s scope. See the
library manual or follow-up texts for details.
struct module — you will need to
understand 3.X’s new bytes object
and 3.X’s different and sharper distinction between text and binary
data and files.str string
type, text files, and all the familiar string operations we studied
earlier. Your strings will be encoded and decoded by 3.X using your
platform’s default encoding (e.g., ASCII, or UTF-8 on Windows in the
U.S. — sys.getdefaultencoding gives
your default if you care to check), but you probably won’t
notice.>>>ord('a')# 'a' is a byte coded as value 97 in ASCII (and others)97 >>>hex(97)'0x61' >>>chr(97)# Code value 97 stands for character 'a' in ASCII'a'
>>>0xC4196 >>>chr(196)# Python 3.X result form shown'Ä'
>>>S = 'ni'>>>S.encode('ascii'), S.encode('latin1'), S.encode('utf8')(b'ni', b'ni', b'ni') >>>S.encode('utf16'), len(S.encode('utf16'))(b'\xff\xfen\x00i\x00', 6) >>>S.encode('utf32'), len(S.encode('utf32'))(b'\xff\xfe\x00\x00n\x00\x00\x00i\x00\x00\x00', 12)
Python 3.3 and later instead use a variable-length scheme with 1, 2, or 4 bytes per character, depending on a string’s content. The size is chosen based upon the character with the largest Unicode ordinal value in the represented string. This scheme allows a space-efficient representation in common cases, but also allows for full UCS-4 on all platforms.
Today, both string content and length really correspond to Unicode code points — identifying ordinal numbers for characters. For instance, the built-inordfunction now returns a character’s Unicode code point ordinal, which is not necessarily an ASCII code, and which may or may not fit in a single 8-bit byte’s value. Similarly,lenreturns the number of characters, not bytes; the string is probably larger in memory, and its characters may not fit in bytes anyhow.
As we saw by example in Chapter 4, under Unicode a single character does not necessarily map directly to a single byte, either when encoded in a file or when stored in memory. Even characters in simple 7-bit ASCII text may not map to bytes — UTF-16 uses multiple bytes per character in files, and Python may allocate 1, 2, or 4 bytes per character in memory. Thinking in terms of characters allows us to abstract away the details of external and internal storage.
When a file is opened in text mode, reading its data automatically decodes its content and returns it as astr; writing takes astrand automatically encodes it before transferring it to the file. Both reads and writes translate per a platform default or a provided encoding name. Text-mode files also support universal end-of-line translation and additional encoding specification arguments. Depending on the encoding name, text files may also automatically process the byte order mark sequence at the start of a file (more on this momentarily).
When a file is opened in binary mode by adding ab(lowercase only) to the mode-string argument in the built-inopencall, reading its data does not decode it in any way but simply returns its content raw and unchanged, as abytesobject; writing similarly takes abytesobject and transfers it to the file unchanged. Binary-mode files also accept abytearrayobject for the content to be written to the file.
bytes and
binary-mode files. You might also opt for
bytearray if you wish to update
the data without making copies of it in memory.str and text-mode
files.C:\code>C:\python33\python>>>B = b'spam'# 3.X bytes literal make a bytes object (8-bit bytes) >>>S = 'eggs'# 3.X str literal makes a Unicode text string>>>type(B), type(S)(<class 'bytes'>, <class 'str'>) >>>B# bytes: sequence of int, prints as character stringb'spam' >>>S'eggs'
>>>B[0], S[0]# Indexing returns an int for bytes, str for str(115, 'e') >>>B[1:], S[1:]# Slicing makes another bytes or str object(b'pam', 'ggs') >>>list(B), list(S)([115, 112, 97, 109], ['e', 'g', 'g', 's'])# bytes is really 8-bit small ints
>>>B[0] = 'x'# Both are immutableTypeError: 'bytes' object does not support item assignment >>>S[0] = 'x'TypeError: 'str' object does not support item assignment
>>># bytes prefix works on single, double, triple quotes, raw>>>B = B"""...xxxx...yyyy...""">>>Bb'\nxxxx\nyyyy\n'
C:\code>C:\python33\python>>>U = u'spam'# 2.X Unicode literal accepted in 3.3+>>>type(U)# It is just str, but is backward compatible<class 'str'> >>>U'spam' >>>U[0]'s' >>>list(U)['s', 'p', 'a', 'm']
C:\code>C:\python27\python>>>B = b'spam'# 3.X bytes literal is just str in 2.6/2.7>>>S = 'eggs'# str is a bytes/character sequence>>>type(B), type(S)(<type 'str'>, <type 'str'>) >>>B, S('spam', 'eggs') >>>B[0], S[0]('s', 'e') >>>list(B), list(S)(['s', 'p', 'a', 'm'], ['e', 'g', 'g', 's'])
>>>U = u'spam'# 2.X Unicode literal makes a distinct type>>>type(U)# Works in 3.3 too, but is just a str there<type 'unicode'> >>>Uu'spam' >>>U[0]u's' >>>list(U)[u's', u'p', u'a', u'm']
str.encode() and bytes(S, encoding) translate a string to its raw bytes form and create an encoded
bytes from a decoded str in the process.bytes.decode() and str(B, encoding) translate raw bytes into its string form and create a decoded
str from an encoded bytes in the process.>>>S = 'eggs'>>>S.encode()# str->bytes: encode text into raw bytesb'eggs' >>>bytes(S, encoding='ascii')# str->bytes, alternativeb'eggs' >>>B = b'spam'>>>B.decode()# bytes->str: decode raw bytes into text'spam' >>>str(B, encoding='ascii')# bytes->str, alternative'spam'
>>>import sys>>>sys.platform# Underlying platform'win32' >>>sys.getdefaultencoding()# Default encoding for str here'utf-8' >>>bytes(S)TypeError: string argument without an encoding >>>str(B)# str without encoding"b'spam'"# A print string, not conversion!>>>len(str(B))7 >>>len(str(B, encoding='ascii'))# Use encoding to convert to str4
>>>S = 'spam'# 2.X type string conversion tools>>>U = u'eggs'>>>S, U('spam', u'eggs') >>>unicode(S), str(U)# 2.X converts str->uni, uni->str(u'spam', 'eggs') >>>S.decode(), U.encode()# versus 3.X byte->str, str->bytes(u'spam', 'eggs')
C:\code>C:\python33\python>>>ord('X')# 'X' is binary code point value 88 in the default encoding88 >>>chr(88)# 88 stands for character 'X''X' >>>S = 'XYZ'# A Unicode string of ASCII text>>>S'XYZ' >>>len(S)# Three characters long3 >>>[ord(c) for c in S]# Three characters with integer ordinal values[88, 89, 90]
>>>S.encode('ascii')# Values 0..127 in 1 byte (7 bits) eachb'XYZ' >>>S.encode('latin-1')# Values 0..255 in 1 byte (8 bits) eachb'XYZ' >>>S.encode('utf-8')# Values 0..127 in 1 byte, 128..2047 in 2, others 3 or 4b'XYZ'
>>>S.encode('latin-1')b'XYZ' >>>S.encode('latin-1')[0]88 >>>list(S.encode('latin-1'))[88, 89, 90]
>>>chr(0xc4)# 0xC4, 0xE8: characters outside ASCII's range'Ä' >>>chr(0xe8)'è' >>>S = '\xc4\xe8'# Single 8-bit value hex escapes: two digits>>>S'Äè' >>>S = '\u00c4\u00e8'# 16-bit Unicode escapes: four digits each>>>S'Äè' >>>len(S)# Two characters long (not number of bytes!)2
>>>S = '\U000000c4\U000000e8'# 32-bit Unicode escapes: eight digits each>>>S'Äè'
>>>S = '\u00c4\u00e8'# Non-ASCII text string, two characters long>>>S'Äè' >>>len(S)2 >>>S.encode('ascii')UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)
>>>S.encode('latin-1')# 1 byte per character when encodedb'\xc4\xe8' >>>S.encode('utf-8')# 2 bytes per character when encodedb'\xc3\x84\xc3\xa8' >>>len(S.encode('latin-1'))# 2 bytes in latin-1, 4 in utf-82 >>>len(S.encode('utf-8'))4
>>>B = b'\xc4\xe8'# Text encoded per Latin-1>>>Bb'\xc4\xe8' >>>len(B)# 2 raw bytes, two encoded characters2 >>>B.decode('latin-1')# Decode to text per Latin-1'Äè' >>>B = b'\xc3\x84\xc3\xa8'# Text encoded per UTF-8>>>len(B)# 4 raw bytes, two encoded characters4 >>>B.decode('utf-8')# Decode to text per UTF-8'Äè' >>>len(B.decode('utf-8'))# Two Unicode characters in memory2
>>>S = 'A\u00c4B\U000000e8C'>>>S# A, B, C, and 2 non-ASCII characters'AÄBèC' >>>len(S)# Five characters long5 >>>S.encode('latin-1')b'A\xc4B\xe8C' >>>len(S.encode('latin-1'))# 5 bytes when encoded per latin-15 >>>S.encode('utf-8')b'A\xc3\x84B\xc3\xa8C' >>>len(S.encode('utf-8'))# 7 bytes when encoded per utf-87
>>>S = 'A' + chr(0xC4) + 'B' + chr(0xE8) + 'C'>>>S'AÄBèC'
>>>S'AÄBèC' >>>S.encode('cp500')# Two other Western European encodingsb'\xc1c\xc2T\xc3' >>>S.encode('cp850')# 5 bytes each, different encoded valuesb'A\x8eB\x8aC' >>>S = 'spam'# ASCII text is the same in most>>>S.encode('latin-1')b'spam' >>>S.encode('utf-8')b'spam' >>>S.encode('cp500')# But not in cp500: IBM EBCDIC!b'\xa2\x97\x81\x94' >>>S.encode('cp850')b'spam'
>>>S = 'A\u00c4B\U000000e8C'>>>S.encode('utf-16')b'\xff\xfeA\x00\xc4\x00B\x00\xe8\x00C\x00' >>>S = 'spam'>>>S.encode('utf-16')b'\xff\xfes\x00p\x00a\x00m\x00' >>>S.encode('utf-32')b'\xff\xfe\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00'
>>>S = 'A\xC4B\xE8C'# 3.X: str recognizes hex and Unicode escapes>>>S'AÄBèC' >>>S = 'A\u00C4B\U000000E8C'>>>S'AÄBèC' >>>B = b'A\xC4B\xE8C'# bytes recognizes hex but not Unicode>>>Bb'A\xc4B\xe8C' >>>B = b'A\u00C4B\U000000E8C'# Escape sequences taken literally!>>>Bb'A\\u00C4B\\U000000E8C' >>>B = b'A\xC4B\xE8C'# Use hex escapes for bytes>>>B# Prints non-ASCII as hexb'A\xc4B\xe8C' >>>print(B)b'A\xc4B\xe8C' >>>B.decode('latin-1')# Decode as latin-1 to interpret as text'AÄBèC'
>>>S = 'AÄBèC'# Chars from UTF-8 if no encoding declaration>>>S'AÄBèC' >>>B = b'AÄBèC'SyntaxError: bytes can only contain ASCII literal characters. >>>B = b'A\xC4B\xE8C'# Chars must be ASCII, or escapes>>>Bb'A\xc4B\xe8C' >>>B.decode('latin-1')'AÄBèC' >>>S.encode()# Source code encoded per UTF-8 by defaultb'A\xc3\x84B\xc3\xa8C'# Uses system default to encode, unless passed>>>S.encode('utf-8')b'A\xc3\x84B\xc3\xa8C' >>>B.decode()# Raw bytes do not correspond to utf-8UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: ...
>>>B = b'A\xc3\x84B\xc3\xa8C'# Text encoded in UTF-8 format originally>>>S = B.decode('utf-8')# Decode to Unicode text per UTF-8>>>S'AÄBèC' >>>T = S.encode('cp500')# Convert to encoded bytes per EBCDIC>>>Tb'\xc1c\xc2T\xc3' >>>U = T.decode('cp500')# Convert back to Unicode per EBCDIC>>>U'AÄBèC' >>>U.encode()# Per default utf-8 encoding againb'A\xc3\x84B\xc3\xa8C'
C:\code>C:\python27\python>>>S = 'A\xC4B\xE8C'# String of 8-bit bytes>>>S# Text encoded per Latin-1, some non-ASCII'A\xc4B\xe8C' >>>print S# Nonprintable characters (IDLE may differ)A─BΦC >>>U = S.decode('latin1')# Decode bytes to Unicode text per latin-1>>>Uu'A\xc4B\xe8C' >>>print UAÄBèC >>>S.decode('utf-8')# Encoded form not compatible with utf-8UnicodeDecodeError: 'utf8' codec can't decode byte 0xc4 in position 1: invalid c ontinuation byte >>>S.decode('ascii')# Encoded bytes are also outside ASCII rangeUnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 1: ordinal not in range(128)
>>>U = u'A\xC4B\xE8C'# Make Unicode string, hex escapes>>>Uu'A\xc4B\xe8C' >>>print UAÄBèC
>>>U.encode('latin-1')# Encode per latin-1: 8-bit bytes'A\xc4B\xe8C' >>>U.encode('utf-8')# Encode per utf-8: multibyte'A\xc3\x84B\xc3\xa8C'
C:\code>C:\python27\python>>>U = u'A\xC4B\xE8C'# Hex escapes for non-ASCII>>>Uu'A\xc4B\xe8C' >>>print UAÄBèC >>>U = u'A\u00C4B\U000000E8C'# Unicode escapes for non-ASCII>>>U# u'' = 16 bits, U'' = 32 bitsu'A\xc4B\xe8C' >>>print UAÄBèC >>>S = 'A\xC4B\xE8C'# Hex escapes work>>>S'A\xc4B\xe8C' >>>print S# But some may print oddly, unless decodedA─BΦC >>>print S.decode('latin-1')AÄBèC >>>S = 'A\u00C4B\U000000E8C'# Not Unicode escapes: taken literally!>>>S'A\\u00C4B\\U000000E8C' >>>print SA\u00C4B\U000000E8C >>>len(S)19
>>>u'ab' + 'cd'# Can mix if compatible in 2.Xu'abcd'# But 'ab' + b'cd' not allowed in 3.X
>>>S = 'A\xC4B\xE8C'# Can't mix in 2.X if str is non-ASCII!>>>U = u'A\xC4B\xE8C'>>>S + UUnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 1: ordinal not in range(128) >>>'abc' + U# Can mix only if str is all 7-bit ASCIIu'abcA\xc4B\xe8C' >>>print 'abc' + U# Use print to display charactersabcAÄBèC >>>S.decode('latin-1') + U# Manual conversion may be required in 2.X toou'A\xc4B\xe8CA\xc4B\xe8C' >>>print S.decode('latin-1') + UAÄBèCAÄBèC >>>print u'\xA3' + '999.99'# Also see Chapter 25's currency example£999.99
>>>str(u'spam')# Unicode to normal'spam' >>>unicode('spam')# Normal to Unicodeu'spam'
# -*- coding: latin-1 -*-
# -*- coding: latin-1 -*-
# Any of the following string literal forms work in latin-1.
# Changing the encoding above to either ascii or utf-8 fails,
# because the 0xc4 and 0xe8 in myStr1 are not valid in either.
myStr1 = 'aÄBèC'
myStr2 = 'A\u00c4B\U000000e8C'
myStr3 = 'A' + chr(0xC4) + 'B' + chr(0xE8) + 'C'
import sys
print('Default encoding:', sys.getdefaultencoding())
for aStr in myStr1, myStr2, myStr3:
print('{0}, strlen={1}, '.format(aStr, len(aStr)), end='')
bytes1 = aStr.encode() # Per default utf-8: 2 bytes for non-ASCII
bytes2 = aStr.encode('latin-1') # One byte per char
#bytes3 = aStr.encode('ascii') # ASCII fails: outside 0..127 range
print('byteslen1={0}, byteslen2={1}'.format(len(bytes1), len(bytes2)))C:\code> C:\python33\python text.py
Default encoding: utf-8
aÄBèC, strlen=5, byteslen1=7, byteslen2=5
AÄBèC, strlen=5, byteslen1=7, byteslen2=5
AÄBèC, strlen=5, byteslen1=7, byteslen2=5C:\code>C:\python33\python# Attributes in str but not bytes>>>set(dir('abc')) - set(dir(b'abc')){'isdecimal', '__mod__', '__rmod__', 'format_map', 'isprintable', 'casefold', 'format', 'isnumeric', 'isidentifier', 'encode'}# Attributes in bytes but not str>>>set(dir(b'abc')) - set(dir('abc')){'decode', 'fromhex'}
>>>B = b'spam'# b'...' bytes literal>>>B.find(b'pa')1 >>>B.replace(b'pa', b'XY')# bytes methods expect bytes argumentsb'sXYm' >>>B.split(b'pa')# bytes methods return bytes results[b's', b'm'] >>>Bb'spam' >>>B[0] = 'x'TypeError: 'bytes' object does not support item assignment
>>>'%s' % 99'99' >>>b'%s' % 99TypeError: unsupported operand type(s) for %: 'bytes' and 'int' >>>'{0}'.format(99)'99' >>>b'{0}'.format(99)AttributeError: 'bytes' object has no attribute 'format'
>>>B = b'spam'# A sequence of small ints>>>B# Prints as ASCII characters (and/or hex escapes)b'spam' >>>B[0]# Indexing yields an int115 >>>B[-1]109 >>>chr(B[0])# Show character for int's' >>>list(B)# Show all the byte's int values[115, 112, 97, 109] >>>B[1:], B[:-1](b'pam', b'spa') >>>len(B)4 >>>B + b'lmn'b'spamlmn' >>>B * 4b'spamspamspamspam'
>>>B = b'abc'# Literal>>>Bb'abc' >>>B = bytes('abc', 'ascii')# Constructor with encoding name>>>Bb'abc' >>>ord('a')97 >>>B = bytes([97, 98, 99])# Integer iterable>>>Bb'abc' >>>B = 'spam'.encode()# str.encode() (or bytes())>>>Bb'spam' >>> >>>S = B.decode()# bytes.decode() (or str())>>>S'spam'
# Must pass expected types to function and method calls>>>B = b'spam'>>>B.replace('pa', 'XY')TypeError: expected an object with the buffer interface >>>B.replace(b'pa', b'XY')b'sXYm' >>>B = B'spam'>>>B.replace(bytes('pa'), bytes('xy'))TypeError: string argument without an encoding >>>B.replace(bytes('pa', 'ascii'), bytes('xy', 'utf-8'))b'sxym'# Must convert manually in 3.X mixed-type expressions>>>b'ab' + 'cd'TypeError: can't concat bytes to str >>>b'ab'.decode() + 'cd'# bytes to str'abcd' >>>b'ab' + 'cd'.encode()# str to bytesb'abcd' >>>b'ab' + bytes('cd', 'ascii')# str to bytesb'abcd'
# Creation in 2.6/2.7: a mutable sequence of small (0..255) ints>>>S = 'spam'>>>C = bytearray(S)# A back-port from 3.X in 2.6+>>>C# b'..' == '..' in 2.6+ (str)bytearray(b'spam')
# Creation in 3.X: text/binary do not mix>>>S = 'spam'>>>C = bytearray(S)TypeError: string argument without an encoding >>>C = bytearray(S, 'latin1')# A content-specific type in 3.X>>>Cbytearray(b'spam') >>>B = b'spam'# b'..' != '..' in 3.X (bytes/str)>>>C = bytearray(B)>>>Cbytearray(b'spam')
# Mutable, but must assign ints, not strings>>>C[0]115 >>>C[0] = 'x'# This and the next work in 2.6/2.7TypeError: an integer is required >>>C[0] = b'x'TypeError: an integer is required >>>C[0] = ord('x')# Use ord() to get a character's ordinal>>>Cbytearray(b'xpam') >>>C[1] = b'Y'[0]# Or index a byte string>>>Cbytearray(b'xYam')
# in bytes but not bytearray>>>set(dir(b'abc')) - set(dir(bytearray(b'abc'))){'__getnewargs__'}# in bytearray but not bytes>>>set(dir(bytearray(b'abc'))) - set(dir(b'abc')){'__iadd__', 'reverse', '__setitem__', 'extend', 'copy', '__alloc__', '__delitem__', '__imul__', 'remove', 'clear', 'insert', 'append', 'pop'}
# Mutable method calls>>>Cbytearray(b'xYam') >>>C.append(b'LMN')# 2.X requires string of size 1TypeError: an integer is required >>>C.append(ord('L'))>>>Cbytearray(b'xYamL') >>>C.extend(b'MNO')>>>Cbytearray(b'xYamLMNO')
# Sequence operations and string methods>>>Cbytearray(b'xYamLMNO') >>>C + b'!#'bytearray(b'xYamLMNO!#') >>>C[0]120 >>>C[1:]bytearray(b'YamLMNO') >>>len(C)8 >>>C.replace('xY', 'sp')# This works in 2.XTypeError: Type str doesn't support the buffer API >>>C.replace(b'xY', b'sp')bytearray(b'spamLMNO') >>>Cbytearray(b'xYamLMNO') >>>C * 4bytearray(b'xYamLMNOxYamLMNOxYamLMNOxYamLMNO')
# Binary versus text>>>B# B is same as S in 2.6/2.7b'spam' >>>list(B)[115, 112, 97, 109] >>>Cbytearray(b'xYamLMNO') >>>list(C)[120, 89, 97, 109, 76, 77, 78, 79] >>>S'spam' >>>list(S)['s', 'p', 'a', 'm']
str for textual
data.bytes for binary
data.bytearray for binary
data you wish to change in place.open, you can force conversions for various
types of Unicode files. Text-mode files also perform universal
line-end translations: by default, all line-end
forms map to the single '\n'
character in your script, regardless of the platform on which you run
it. As described earlier, text files also handle reading and writing
the byte order mark (BOM) stored at the
start-of-file in some Unicode encoding schemes.C:\code>C:\python33\python# Basic text files (and strings) work the same as in 2.X>>>file = open('temp', 'w')>>>size = file.write('abc\n')# Returns number of characters written>>>file.close()# Manual close to flush output buffer>>>file = open('temp')# Default mode is "r" (== "rt"): text input>>>text = file.read()>>>text'abc\n' >>>print(text)abc
C:\code>C:\python27\python>>>open('temp', 'w').write('abd\n')# Write in text mode: adds \r>>>open('temp', 'r').read()# Read in text mode: drops \r'abd\n' >>>open('temp', 'rb').read()# Read in binary mode: verbatim'abd\r\n' >>>open('temp', 'wb').write('abc\n')# Write in binary mode>>>open('temp', 'r').read()# \n not expanded to \r\n'abc\n' >>>open('temp', 'rb').read()'abc\n'
C:\code>C:\python33\python# Write and read a text file>>>open('temp', 'w').write('abc\n')# Text mode output, provide a str4 >>>open('temp', 'r').read()# Text mode input, returns a str'abc\n' >>>open('temp', 'rb').read()# Binary mode input, returns a bytesb'abc\r\n'
# Write and read a binary file>>>open('temp', 'wb').write(b'abc\n')# Binary mode output, provide a bytes4 >>>open('temp', 'r').read()# Text mode input, returns a str'abc\n' >>>open('temp', 'rb').read()# Binary mode input, returns a bytesb'abc\n'
# Write and read truly binary data>>>open('temp', 'wb').write(b'a\x00c')# Provide a bytes3 >>>open('temp', 'r').read()# Receive a str'a\x00c' >>>open('temp', 'rb').read()# Receive a bytesb'a\x00c'
# bytearrays work too >>>BA = bytearray(b'\x01\x02\x03')>>>open('temp', 'wb').write(BA)3 >>>open('temp', 'r').read()'\x01\x02\x03' >>>open('temp', 'rb').read()b'\x01\x02\x03'
# Types are not flexible for file content>>>open('temp', 'w').write('abc\n')# Text mode makes and requires str4 >>>open('temp', 'w').write(b'abc\n')TypeError: must be str, not bytes >>>open('temp', 'wb').write(b'abc\n')# Binary mode makes and requires bytes4 >>>open('temp', 'wb').write('abc\n')TypeError: 'str' does not support the buffer interface
# Can't read truly binary data in text mode>>>chr(0xFF)# FF is a valid char, FE is not'ÿ' >>>chr(0xFE)# An error in some Pythons'\xfe' >>>open('temp', 'w').write(b'\xFF\xFE\xFD')# Can't use arbitrary bytes!TypeError: must be str, not bytes >>>open('temp', 'w').write('\xFF\xFE\xFD')# Can write if embeddable in str3 >>>open('temp', 'wb').write(b'\xFF\xFE\xFD')# Can also write in binary mode3 >>>open('temp', 'rb').read()# Can always read as binary bytesb'\xff\xfe\xfd' >>>open('temp', 'r').read()# Can't read text unless decodable!'ÿ\xfe\xfd'# An error in some Pythons
C:\code>C:\python33\python>>>S = 'A\xc4B\xe8C'# Five-character decoded string, non-ASCII>>>S'AÄBèC' >>>len(S)5
# Encode manually with methods>>>L = S.encode('latin-1')# 5 bytes when encoded as latin-1>>>Lb'A\xc4B\xe8C' >>>len(L)5 >>>U = S.encode('utf-8')# 7 bytes when encoded as utf-8>>>Ub'A\xc3\x84B\xc3\xa8C' >>>len(U)7
# Encoding automatically when written>>>open('latindata', 'w', encoding='latin-1').write(S)# Write as latin-15 >>>open('utf8data', 'w', encoding='utf-8').write(S)# Write as utf-85 >>>open('latindata', 'rb').read()# Read raw bytesb'A\xc4B\xe8C' >>>open('utf8data', 'rb').read()# Different in filesb'A\xc3\x84B\xc3\xa8C'
# Decoding automatically when read>>>open('latindata', 'r', encoding='latin-1').read()# Decoded on input'AÄBèC' >>>open('utf8data', 'r', encoding='utf-8').read()# Per encoding type'AÄBèC' >>>X = open('latindata', 'rb').read()# Manual decoding:>>>X.decode('latin-1')# Not necessary'AÄBèC' >>>X = open('utf8data', 'rb').read()>>>X.decode()# UTF-8 is default'AÄBèC'
>>>file = open(r'C:\Python33\python.exe', 'r')>>>text = file.read()UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 2: ... >>>file = open(r'C:\Python33\python.exe', 'rb')>>>data = file.read()>>>data[:20]b'MZ\x90\x00\x03\x00\x00\x00\x04\x00\x00\x00\xff\xff\x00\x00\xb8\x00\x00\x00'
C:\code>C:\python33\python# File saved in Notepad>>>import sys>>>sys.getdefaultencoding()'utf-8' >>>open('spam.txt', 'rb').read()# ASCII (UTF-8) text fileb'spam\r\nSPAM\r\n' >>>open('spam.txt', 'r').read()# Text mode translates line end'spam\nSPAM\n' >>>open('spam.txt', 'r', encoding='utf-8').read()'spam\nSPAM\n'
>>>open('spam.txt', 'rb').read()# UTF-8 with 3-byte BOMb'\xef\xbb\xbfspam\r\nSPAM\r\n' >>>open('spam.txt', 'r').read()'spam\nSPAM\n' >>>open('spam.txt', 'r', encoding='utf-8').read()'\ufeffspam\nSPAM\n' >>>open('spam.txt', 'r', encoding='utf-8-sig').read()'spam\nSPAM\n'
>>>open('spam.txt', 'rb').read()b'\xfe\xff\x00s\x00p\x00a\x00m\x00\r\x00\n\x00S\x00P\x00A\x00M\x00\r\x00\n' >>>open('spam.txt', 'r').read()'\xfeÿ\x00s\x00p\x00a\x00m\x00\n\x00\n\x00S\x00P\x00A\x00M\x00\n\x00\n' >>>open('spam.txt', 'r', encoding='utf-16').read()'spam\nSPAM\n' >>>open('spam.txt', 'r', encoding='utf-16-be').read()'\ufeffspam\nSPAM\n'
>>>open('temp.txt', 'w', encoding='utf-8').write('spam\nSPAM\n')10 >>>open('temp.txt', 'rb').read()# No BOMb'spam\r\nSPAM\r\n' >>>open('temp.txt', 'w', encoding='utf-8-sig').write('spam\nSPAM\n')10 >>>open('temp.txt', 'rb').read()# Wrote BOMb'\xef\xbb\xbfspam\r\nSPAM\r\n' >>>open('temp.txt', 'r').read()'spam\nSPAM\n' >>>open('temp.txt', 'r', encoding='utf-8').read()# Keeps BOM'\ufeffspam\nSPAM\n' >>>open('temp.txt', 'r', encoding='utf-8-sig').read()# Skips BOM'spam\nSPAM\n'
>>>open('temp.txt', 'w').write('spam\nSPAM\n')10 >>>open('temp.txt', 'rb').read()# Data without BOMb'spam\r\nSPAM\r\n' >>>open('temp.txt', 'r').read()# Either utf-8 works'spam\nSPAM\n' >>>open('temp.txt', 'r', encoding='utf-8').read()'spam\nSPAM\n' >>>open('temp.txt', 'r', encoding='utf-8-sig').read()'spam\nSPAM\n'
>>>sys.byteorder'little' >>>open('temp.txt', 'w', encoding='utf-16').write('spam\nSPAM\n')10 >>>open('temp.txt', 'rb').read()b'\xff\xfes\x00p\x00a\x00m\x00\r\x00\n\x00S\x00P\x00A\x00M\x00\r\x00\n\x00' >>>open('temp.txt', 'r', encoding='utf-16').read()'spam\nSPAM\n'
>>>open('temp.txt', 'w', encoding='utf-16-be').write('\ufeffspam\nSPAM\n')11 >>>open('spam.txt', 'rb').read()b'\xfe\xff\x00s\x00p\x00a\x00m\x00\r\x00\n\x00S\x00P\x00A\x00M\x00\r\x00\n' >>>open('temp.txt', 'r', encoding='utf-16').read()'spam\nSPAM\n' >>>open('temp.txt', 'r', encoding='utf-16-be').read()'\ufeffspam\nSPAM\n'
>>>open('temp.txt', 'w', encoding='utf-16-le').write('SPAM')4 >>>open('temp.txt', 'rb').read()# OK if BOM not present or expectedb'S\x00P\x00A\x00M\x00' >>>open('temp.txt', 'r', encoding='utf-16-le').read()'SPAM' >>>open('temp.txt', 'r', encoding='utf-16').read()UnicodeError: UTF-16 stream does not start with BOM
C:\code>C:\python27\python>>>S = u'A\xc4B\xe8C'# 2.X type>>>print SAÄBèC >>>len(S)5 >>>S.encode('latin-1')# Manual calls'A\xc4B\xe8C' >>>S.encode('utf-8')'A\xc3\x84B\xc3\xa8C' >>>import codecs# 2.X files>>>codecs.open('latindata', 'w', encoding='latin-1').write(S)# Writes encode>>>codecs.open('utfdata', 'w', encoding='utf-8').write(S)>>>open('latindata', 'rb').read()'A\xc4B\xe8C' >>>open('utfdata', 'rb').read()'A\xc3\x84B\xc3\xa8C' >>>codecs.open('latindata', 'r', encoding='latin-1').read()# Reads decodeu'A\xc4B\xe8C' >>>codecs.open('utfdata', 'r', encoding='utf-8').read()u'A\xc4B\xe8C' >>>print codecs.open('utfdata', 'r', encoding='utf-8').read()# Print to viewAÄBèC
>>>import sys>>>sys.getdefaultencoding(), sys.getfilesystemencoding()# File content, names('utf-8', 'mbcs')
>>>f = open('xxx\u00A5', 'w')# Non-ASCII filename>>>f.write('\xA5999\n')# Writes five characters>>>f.close()>>>print(open('xxx\u00A5').read())# Text: auto-encoded¥999 >>>print(open(b'xxx\xA5').read())# Bytes: pre-encoded¥999 >>>import glob# Filename expansion tool>>>glob.glob('*\u00A5*')# Get decoded text for decoded text['xxx¥'] >>>glob.glob(b'*\xA5*')# Get encoded bytes for encoded bytes[b'xxx\xa5']
C:\code>C:\python33\python>>>import re>>>S = 'Bugger all down here on earth!'# Line of text>>>B = b'Bugger all down here on earth!'# Usually from a file>>>re.match('(.*) down (.*) on (.*)', S).groups()# Match line to pattern('Bugger all', 'here', 'earth!')# Matched substrings>>>re.match(b'(.*) down (.*) on (.*)', B).groups()# bytes substrings(b'Bugger all', b'here', b'earth!')
C:\code>C:\python27\python>>>import re>>>S = 'Bugger all down here on earth!'# Simple text and binary>>>U = u'Bugger all down here on earth!'# Unicode text>>>re.match('(.*) down (.*) on (.*)', S).groups()('Bugger all', 'here', 'earth!') >>>re.match('(.*) down (.*) on (.*)', U).groups()(u'Bugger all', u'here', u'earth!')
C:\code>C:\python33\python>>>import re>>>S = 'Bugger all down here on earth!'>>>B = b'Bugger all down here on earth!'>>>re.match('(.*) down (.*) on (.*)', B).groups()TypeError: can't use a string pattern on a bytes-like object >>>re.match(b'(.*) down (.*) on (.*)', S).groups()TypeError: can't use a bytes pattern on a string-like object >>>re.match(b'(.*) down (.*) on (.*)', bytearray(B)).groups()(bytearray(b'Bugger all'), bytearray(b'here'), bytearray(b'earth!')) >>>re.match('(.*) down (.*) on (.*)', bytearray(B)).groups()TypeError: can't use a string pattern on a bytes-like object
C:\code>C:\python33\python>>>from struct import pack>>>pack('>i4sh', 7, b'spam', 8)# bytes in 3.X (8-bit strings)b'\x00\x00\x00\x07spam\x00\x08' C:\code>C:\python27\python>>>from struct import pack>>>pack('>i4sh', 7, 'spam', 8)# str in 2.X (8-bit strings)'\x00\x00\x00\x07spam\x00\x08'
C:\code>C:\python33\python>>>import struct>>>B = struct.pack('>i4sh', 7, b'spam', 8)>>>Bb'\x00\x00\x00\x07spam\x00\x08' >>>vals = struct.unpack('>i4sh', B)>>>vals(7, b'spam', 8) >>>vals = struct.unpack('>i4sh', B.decode())TypeError: 'str' does not support the buffer interface
C:\code>C:\python33\python# Write values to a packed binary file>>>F = open('data.bin', 'wb')# Open binary output file>>>import struct>>>data = struct.pack('>i4sh', 7, b'spam', 8)# Create packed binary data>>>data# bytes in 3.X, not strb'\x00\x00\x00\x07spam\x00\x08' >>>F.write(data)# Write to the file10 >>>F.close()# Read values from a packed binary file>>>F = open('data.bin', 'rb')# Open binary input file>>>data = F.read()# Read bytes>>>datab'\x00\x00\x00\x07spam\x00\x08' >>>values = struct.unpack('>i4sh', data)# Extract packed binary data>>>values# Back to Python objects(7, b'spam', 8)
>>>values# Result of struct.unpack(7, b'spam', 8)# Accessing bits of parsed integers>>>bin(values[0])# Can get to bits in ints'0b111' >>>values[0] & 0x01# Test first (lowest) bit in int1 >>>values[0] | 0b1010# Bitwise or: turn bits on15 >>>bin(values[0] | 0b1010)# 15 decimal is 1111 binary'0b1111' >>>bin(values[0] ^ 0b1010)# Bitwise xor: off if both true'0b1101' >>>bool(values[0] & 0b100)# Test if bit 3 is onTrue >>>bool(values[0] & 0b1000)# Test if bit 4 is setFalse
# Accessing bytes of parsed strings and bits within them>>>values[1]b'spam' >>>values[1][0]# bytes string: sequence of ints115 >>>values[1][1:]# Prints as ASCII charactersb'pam' >>>bin(values[1][0])# Can get to bits of bytes in strings'0b1110011' >>>bin(values[1][0] | 0b1100)# Turn bits on'0b1111111' >>>values[1][0] | 0b1100127
C:\code>C:\python33\python>>>import pickle# dumps() returns pickle string>>>pickle.dumps([1, 2, 3])# Python 3.X default protocol=3=binaryb'\x80\x03]q\x00(K\x01K\x02K\x03e.' >>>pickle.dumps([1, 2, 3], protocol=0)# ASCII protocol 0, but still bytes!b'(lp0\nL1L\naL2L\naL3L\na.'
>>>pickle.dump([1, 2, 3], open('temp', 'w'))# Text files fail on bytes!TypeError: must be str, not bytes# Despite protocol value>>>pickle.dump([1, 2, 3], open('temp', 'w'), protocol=0)TypeError: must be str, not bytes >>>pickle.dump([1, 2, 3], open('temp', 'wb'))# Always use binary in 3.X>>>open('temp', 'r').read()# This works, but just by luck'\u20ac\x03]q\x00(K\x01K\x02K\x03e.'
>>>pickle.dump([1, 2, 3], open('temp', 'wb'))>>>pickle.load(open('temp', 'rb'))[1, 2, 3] >>>open('temp', 'rb').read()b'\x80\x03]q\x00(K\x01K\x02K\x03e.'
C:\code>C:\python27\python>>>import pickle>>>pickle.dumps([1, 2, 3])# Python 2.X default=0=ASCII'(lp0\nI1\naI2\naI3\na.' >>>pickle.dumps([1, 2, 3], protocol=1)']q\x00(K\x01K\x02K\x03e.' >>>pickle.dump([1, 2, 3], open('temp', 'w'))# Text mode works in 2.X>>>pickle.load(open('temp'))[1, 2, 3] >>>open('temp').read()'(lp0\nI1\naI2\naI3\na.'
>>>import pickle>>>pickle.dump([1, 2, 3], open('temp', 'wb'))# Version neutral>>>pickle.load(open('temp', 'rb'))# And required in 3.X[1, 2, 3]
<books>
<date>1995~2013</date>
<title>Learning Python</title>
<title>Programming Python</title>
<title>Python Pocket Reference</title>
<publisher>O'Reilly Media</publisher>
</books>Learning Python Programming Python Python Pocket Reference
# File patternparse.py
import re
text = open('mybooks.xml').read()
found = re.findall('<title>(.*)</title>', text)
for title in found: print(title)# File domparse.py
from xml.dom.minidom import parse, Node
xmltree = parse('mybooks.xml')
for node1 in xmltree.getElementsByTagName('title'):
for node2 in node1.childNodes:
if node2.nodeType == Node.TEXT_NODE:
print(node2.data)# File saxparse.py
import xml.sax.handler
class BookHandler(xml.sax.handler.ContentHandler):
def __init__(self):
self.inTitle = False
def startElement(self, name, attributes):
if name == 'title':
self.inTitle = True
def characters(self, data):
if self.inTitle:
print(data)
def endElement(self, name):
if name == 'title':
self.inTitle = False
import xml.sax
parser = xml.sax.make_parser()
handler = BookHandler()
parser.setContentHandler(handler)
parser.parse('mybooks.xml')# File etreeparse.py
from xml.etree.ElementTree import parse
tree = parse('mybooks.xml')
for E in tree.findall('title'):
print(E.text)C:\code>C:\python27\python domparse.pyLearning Python Programming Python Python Pocket Reference C:\code>C:\python33\python domparse.pyLearning Python Programming Python Python Pocket Reference
C:\code>C:\python33\python>>>from xml.dom.minidom import parse, Node>>>xmltree = parse('mybooks.xml')>>>for node in xmltree.getElementsByTagName('title'):for node2 in node.childNodes:if node2.nodeType == Node.TEXT_NODE:node2.data'Learning Python' 'Programming Python' 'Python Pocket Reference' C:\code>C:\python27\python>>>...same code...u'Learning Python' u'Programming Python' u'Python Pocket Reference'
str (for Unicode text, including ASCII),
bytes (for binary data with
absolute byte values), and bytearray (a mutable flavor of bytes). The str type usually represents content stored
on a text file, and the other two types generally represent content
stored on binary files.str (for 8-bit text and binary data) and
unicode (for possibly wider
character Unicode text). The str
type is used for both text and binary file content; unicode is used for text file content that
is generally more complex than 8-bit characters. Python 2.6 (but not
earlier) also has 3.X’s bytearray
type, but it’s mostly a back-port and doesn’t exhibit the sharp
text/binary distinction that it does in 3.X.str equates to both str and bytes in 3.X, and 3.X’s str equates to both str and unicode in 2.X. The mutability of bytearray in 3.X is also unique. In general,
though: Unicode text is handled by 3.X str and 2.X unicode, byte-based data is handled by 3.X
bytes and 2.X str, and 3.X bytes and 2.X str can both handle some simpler types of
text.str supports string formatting operations,
and bytearray has an additional set
of operations that perform in-place changes. The str and bytes types also have methods for encoding
and decoding text, respectively.\xNN) and Unicode (\uNNNN, \UNNNNNNNN) escapes. On some machines, some
non-ASCII characters — certain Latin-1 characters, for example — can also
be typed or pasted directly into code, and are interpreted per the
UTF-8 default or a source code encoding directive comment.str objects
in your script, and the contents of binary files are represented as
bytes (or bytearray) objects. Text-mode files also
handle the BOM for certain encoding types and automatically translate
end-of-line sequences to and from the single \n character on input and output unless this
is explicitly disabled; binary-mode files do not perform either of
these steps. Python 2.X uses codecs.open for Unicode files, which encodes
and decodes similarly; 2.X’s open
only translates line ends in text mode.open built-in in 3.X (codecs.open() in 2.X); data will be decoded
per the specified encoding when it is read from the file. You can also
read in binary mode and manually decode the bytes to a string by
giving an encoding name, but this involves extra work and is somewhat
error-prone for multibyte characters (you may accidentally read a
partial character sequence).open in 3.X (codecs.open() in 2.X); strings will be
encoded per the desired encoding when they are written to the file.
You can also manually encode a string to bytes and write it in binary
mode, but this is usually extra work.str string type
works the same in 2.X and 3.X in this case. Moreover, although
string-related tools in the standard library such as re, struct, pickle, and xml may technically use different types in
3.X than in 2.X, the changes are largely irrelevant to most programs
because 3.X’s str and bytes and 2.X’s str support almost identical interfaces. If
you process Unicode data, the toolset you need has simply moved from
2.X’s unicode and codecs.open() to 3.X’s str and open. If you deal with binary data files,
you’ll need to deal with content as bytes objects; since they have a similar
interface to 2.X strings, though, the impact should again be minimal.
That said, the update of the book Programming
Python for 3.X ran across numerous cases where Unicode’s
mandatory status in 3.X implied changes in standard library APIs — from
networking and GUIs, to databases and email. In general, Unicode will
probably impact most 3.X users eventually.person.name# Fetch attribute valueperson.name =value# Change attribute value
class Person:
def getName(self):
if not valid():
raise TypeError('cannot fetch name')
else:
return self.name.transform()
def setName(self, value):
if not valid(value):
raise TypeError('cannot change name')
else:
self.name = transform(value)
person = Person()
person.getName()
person.setName('value')__getattr__ and
__setattr__ methods, for
routing undefined attribute fetches and all attribute
assignments to generic handler methods.__getattribute__
method, for routing all attribute fetches to a generic handler
method.property built-in, for
routing specific attribute access to get and set handler functions.attribute = property(fget, fset, fdel, doc)
class Person:# Add (object) in 2.Xdef __init__(self, name): self._name = name def getName(self): print('fetch...') return self._name def setName(self, value): print('change...') self._name = value def delName(self): print('remove...') del self._name name = property(getName, setName, delName, "name property docs") bob = Person('Bob Smith')# bob has a managed attributeprint(bob.name)# Runs getNamebob.name = 'Robert Smith'# Runs setNameprint(bob.name) del bob.name# Runs delNameprint('-'*20) sue = Person('Sue Jones')# sue inherits property tooprint(sue.name) print(Person.name.__doc__)# Or help(Person.name)
c:\code> py −3 prop-person.py
fetch...
Bob Smith
change...
fetch...
Robert Smith
remove...
--------------------
fetch...
Sue Jones
name property docsclass Super:
...the original Person class code...
name = property(getName, setName, delName, 'name property docs')
class Person(Super):
pass # Properties are inherited (class attrs)
bob = Person('Bob Smith')
...rest unchanged...class PropSquare:
def __init__(self, start):
self.value = start
def getX(self): # On attr fetch
return self.value ** 2
def setX(self, value): # On attr assign
self.value = value
X = property(getX, setX) # No delete or docs
P = PropSquare(3) # Two instances of class with property
Q = PropSquare(32) # Each has different state information
print(P.X) # 3 ** 2
P.X = 4
print(P.X) # 4 ** 2
print(Q.X) # 32 ** 2 (1024)c:\code> py −3 prop-computed.py
9
16
1024@decorator def func(args): ...
def func(args): ... func = decorator(func)
class Person:
@property
def name(self): ... # Rebinds: name = property(name)class Person:
def name(self): ...
name = property(name)class Person:
def __init__(self, name):
self._name = name
@property
def name(self): # name = property(name)
"name property docs"
print('fetch...')
return self._name
@name.setter
def name(self, value): # name = name.setter(name)
print('change...')
self._name = value
@name.deleter
def name(self): # name = name.deleter(name)
print('remove...')
del self._name
bob = Person('Bob Smith') # bob has a managed attribute
print(bob.name) # Runs name getter (name 1)
bob.name = 'Robert Smith' # Runs name setter (name 2)
print(bob.name)
del bob.name # Runs name deleter (name 3)
print('-'*20)
sue = Person('Sue Jones') # sue inherits property too
print(sue.name)
print(Person.name.__doc__) # Or help(Person.name)c:\code> py −3 prop-person-deco.py
fetch...
Bob Smith
change...
fetch...
Robert Smith
remove...
--------------------
fetch...
Sue Jones
name property docsclass Descriptor:
"docstring goes here"
def __get__(self, instance, owner): ... # Return attr value
def __set__(self, instance, value): ... # Return nothing (None)
def __delete__(self, instance): ... # Return nothing (None)>>>class Descriptor:# Add "(object)" in 2.Xdef __get__(self, instance, owner):print(self, instance, owner, sep='\n')>>>class Subject:# Add "(object)" in 2.Xattr = Descriptor()# Descriptor instance is class attr>>>X = Subject()>>>X.attr<__main__.Descriptor object at 0x0281E690> <__main__.Subject object at 0x028289B0> <class '__main__.Subject'> >>>Subject.attr<__main__.Descriptor object at 0x0281E690> None <class '__main__.Subject'>
X.attr -> Descriptor.__get__(Subject.attr, X, Subject)
>>>class D:def __get__(*args): print('get')>>>class C:a = D()# Attribute a is a descriptor instance>>>X = C()>>>X.a# Runs inherited descriptor __get__get >>>C.aget >>>X.a = 99# Stored on X, hiding C.a!>>>X.a99 >>>list(X.__dict__.keys())['a'] >>>Y = C()>>>Y.a# Y still inherits descriptorget >>>C.aget
>>>class D:def __get__(*args): print('get')def __set__(*args): raise AttributeError('cannot set')>>>class C:a = D()>>>X = C()>>>X.a# Routed to C.a.__get__get >>>X.a = 99# Routed to C.a.__set__AttributeError: cannot set
class Name:# Use (object) in 2.X"name descriptor docs" def __get__(self, instance, owner): print('fetch...') return instance._name def __set__(self, instance, value): print('change...') instance._name = value def __delete__(self, instance): print('remove...') del instance._name class Person:# Use (object) in 2.Xdef __init__(self, name): self._name = name name = Name()# Assign descriptor to attrbob = Person('Bob Smith')# bob has a managed attributeprint(bob.name)# Runs Name.__get__bob.name = 'Robert Smith'# Runs Name.__set__print(bob.name) del bob.name# Runs Name.__delete__print('-'*20) sue = Person('Sue Jones')# sue inherits descriptor tooprint(sue.name) print(Name.__doc__)# Or help(Name)
self is the Name class instance.instance is the Person class instance.owner is the Person class.c:\code> py −3 desc-person.py
fetch...
Bob Smith
change...
fetch...
Robert Smith
remove...
--------------------
fetch...
Sue Jones
name descriptor docs...
class Super:
def __init__(self, name):
self._name = name
name = Name()
class Person(Super): # Descriptors are inherited (class attrs)
pass
...class Person:
def __init__(self, name):
self._name = name
class Name: # Using a nested class
"name descriptor docs"
def __get__(self, instance, owner):
print('fetch...')
return instance._name
def __set__(self, instance, value):
print('change...')
instance._name = value
def __delete__(self, instance):
print('remove...')
del instance._name
name = Name()...
print(Person.Name.__doc__) # Differs: not Name.__doc__ outside classclass DescSquare:
def __init__(self, start): # Each desc has own state
self.value = start
def __get__(self, instance, owner): # On attr fetch
return self.value ** 2
def __set__(self, instance, value): # On attr assign
self.value = value # No delete or docs
class Client1:
X = DescSquare(3) # Assign descriptor instance to class attr
class Client2:
X = DescSquare(32) # Another instance in another client class
# Could also code two instances in same class
c1 = Client1()
c2 = Client2()
print(c1.X) # 3 ** 2
c1.X = 4
print(c1.X) # 4 ** 2
print(c2.X) # 32 ** 2 (1024)c:\code> py −3 desc-computed.py
9
16
1024class DescState:# Use descriptor state, (object) in 2.Xdef __init__(self, value): self.value = value def __get__(self, instance, owner):# On attr fetchprint('DescState get') return self.value * 10 def __set__(self, instance, value):# On attr assignprint('DescState set') self.value = value# Client classclass CalcAttrs: X = DescState(2)# Descriptor class attrY = 3# Class attrdef __init__(self): self.Z = 4# Instance attrobj = CalcAttrs() print(obj.X, obj.Y, obj.Z)# X is computed, others are notobj.X = 5# X assignment is interceptedCalcAttrs.Y = 6# Y reassigned in classobj.Z = 7# Z assigned in instanceprint(obj.X, obj.Y, obj.Z) obj2 = CalcAttrs()# But X uses shared data, like Y!print(obj2.X, obj2.Y, obj2.Z)
c:\code> py −3 desc-state-desc.py
DescState get
20 3 4
DescState set
DescState get
50 6 7
DescState get
50 6 4class InstState:# Using instance state, (object) in 2.Xdef __get__(self, instance, owner): print('InstState get')# Assume set by client classreturn instance._X * 10 def __set__(self, instance, value): print('InstState set') instance._X = value# Client classclass CalcAttrs: X = InstState()# Descriptor class attrY = 3# Class attrdef __init__(self): self._X = 2# Instance attrself.Z = 4# Instance attrobj = CalcAttrs() print(obj.X, obj.Y, obj.Z)# X is computed, others are notobj.X = 5# X assignment is interceptedCalcAttrs.Y = 6# Y reassigned in classobj.Z = 7# Z assigned in instanceprint(obj.X, obj.Y, obj.Z) obj2 = CalcAttrs()# But X differs now, like Z!print(obj2.X, obj2.Y, obj2.Z)
c:\code> py −3 desc-state-inst.py
InstState get
20 3 4
InstState set
InstState get
50 6 7
InstState get
20 6 4>>>class DescBoth:def __init__(self, data):self.data = datadef __get__(self, instance, owner):return '%s, %s' % (self.data, instance.data)def __set__(self, instance, value):instance.data = value>>>class Client:def __init__(self, data):self.data = datamanaged = DescBoth('spam')>>>I = Client('eggs')>>>I.managed# Show both data sources'spam, eggs' >>>I.managed = 'SPAM'# Change instance data>>>I.managed'spam, SPAM'
>>>I.__dict__{'data': 'SPAM'} >>>[x for x in dir(I) if not x.startswith('__')]['data', 'managed'] >>>getattr(I, 'data')'SPAM' >>>getattr(I, 'managed')'spam, SPAM' >>>for attr in (x for x in dir(I) if not x.startswith('__')):print('%s => %s' % (attr, getattr(I, attr)))data => SPAM managed => spam, SPAM
class Property:
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel # Save unbound methods
self.__doc__ = doc # or other callables
def __get__(self, instance, instancetype=None):
if instance is None:
return self
if self.fget is None:
raise AttributeError("can't get attribute")
return self.fget(instance) # Pass instance to self
# in property accessors
def __set__(self, instance, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(instance, value)
def __delete__(self, instance):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(instance)
class Person:
def getName(self): print('getName...')
def setName(self, value): print('setName...')
name = Property(getName, setName) # Use like property()
x = Person()
x.name
x.name = 'Bob'
del x.namec:\code> py −3 prop-desc-equiv.py
getName...
setName...
AttributeError: can't delete attribute__getattr__ is run for
undefined attributes — because it is run only for
attributes not stored on an instance or inherited from one of its
classes, its use is straightforward.__getattribute__ is run for
every attribute — because it is all-inclusive, you
must be cautious when using this method to avoid recursive loops by
passing attribute accesses to a superclass.def __getattr__(self, name):# On undefined attribute fetch [obj.name]def __getattribute__(self, name):# On all attribute fetch [obj.name]def __setattr__(self, name, value):# On all attribute assignment [obj.name=value]def __delattr__(self, name):# On all attribute deletion [del obj.name]
class Catcher:
def __getattr__(self, name):
print('Get: %s' % name)
def __setattr__(self, name, value):
print('Set: %s %s' % (name, value))
X = Catcher()
X.job # Prints "Get: job"
X.pay # Prints "Get: pay"
X.pay = 99 # Prints "Set: pay 99"class Catcher(object):# Need (object) in 2.X onlydef __getattribute__(self, name):# Works same as getattr hereprint('Get: %s' % name)# But prone to loops on general...rest unchanged...
class Wrapper:
def __init__(self, object):
self.wrapped = object # Save object
def __getattr__(self, attrname):
print('Trace: ' + attrname) # Trace fetch
return getattr(self.wrapped, attrname) # Delegate fetch
X = Wrapper([1, 2, 3])
X.append(4) # Prints "Trace: append"
print(X.wrapped) # Prints "[1, 2, 3, 4]" def __getattribute__(self, name):
x = self.other # LOOPS! def __getattribute__(self, name):
x = object.__getattribute__(self, 'other') # Force higher to avoid me def __setattr__(self, name, value):
self.other = value # Recurs (and might LOOP!) def __setattr__(self, name, value):
self.__dict__['other'] = value # Use attr dict to avoid me def __setattr__(self, name, value):
object.__setattr__(self, 'other', value) # Force higher to avoid me def __getattribute__(self, name):
x = self.__dict__['other'] # Loops!class Person:# Portable: 2.X or 3.Xdef __init__(self, name):# On [Person()]self._name = name# Triggers __setattr__!def __getattr__(self, attr):# On [obj.undefined]print('get: ' + attr) if attr == 'name':# Intercept name: not storedreturn self._name# Does not loop: real attrelse:# Others are errorsraise AttributeError(attr) def __setattr__(self, attr, value):# On [obj.any = value]print('set: ' + attr) if attr == 'name': attr = '_name'# Set internal nameself.__dict__[attr] = value# Avoid looping heredef __delattr__(self, attr):# On [del obj.any]print('del: ' + attr) if attr == 'name': attr = '_name'# Avoid looping here toodel self.__dict__[attr]# but much less commonbob = Person('Bob Smith')# bob has a managed attributeprint(bob.name)# Runs __getattr__bob.name = 'Robert Smith'# Runs __setattr__print(bob.name) del bob.name# Runs __delattr__print('-'*20) sue = Person('Sue Jones')# sue inherits property tooprint(sue.name) #print(Person.name.__doc__)# No equivalent here
c:\code> py −3 getattr-person.py
set: _name
get: name
Bob Smith
set: name
get: name
Robert Smith
del: name
--------------------
set: _name
get: name
Sue Jones# Replace __getattr__ with thisdef __getattribute__(self, attr):# On [obj.any]print('get: ' + attr) if attr == 'name':# Intercept all namesattr = '_name'# Map to internal namereturn object.__getattribute__(self, attr)# Avoid looping here
c:\code> py −3 getattribute-person.py
set: _name
get: __dict__
get: name
Bob Smith
set: name
get: __dict__
get: name
Robert Smith
del: name
get: __dict__
--------------------
set: _name
get: __dict__
get: name
Sue Jonesclass AttrSquare:
def __init__(self, start):
self.value = start # Triggers __setattr__!
def __getattr__(self, attr): # On undefined attr fetch
if attr == 'X':
return self.value ** 2 # value is not undefined
else:
raise AttributeError(attr)
def __setattr__(self, attr, value): # On all attr assignments
if attr == 'X':
attr = 'value'
self.__dict__[attr] = value
A = AttrSquare(3) # 2 instances of class with overloading
B = AttrSquare(32) # Each has different state information
print(A.X) # 3 ** 2
A.X = 4
print(A.X) # 4 ** 2
print(B.X) # 32 ** 2 (1024)c:\code> py −3 getattr-computed.py
9
16
1024class AttrSquare:# Add (object) for 2.Xdef __init__(self, start): self.value = start# Triggers __setattr__!def __getattribute__(self, attr):# On all attr fetchesif attr == 'X': return self.value ** 2# Triggers __getattribute__ again!else: return object.__getattribute__(self, attr) def __setattr__(self, attr, value):# On all attr assignmentsif attr == 'X': attr = 'value' object.__setattr__(self, attr, value)
self.value=start inside
the constructor triggers __setattr__self.value inside
__getattribute__ triggers
__getattribute__ again def __getattribute__(self, attr):
if attr == 'X':
return object.__getattribute__(self, 'value') ** 2class GetAttr:
attr1 = 1
def __init__(self):
self.attr2 = 2
def __getattr__(self, attr): # On undefined attrs only
print('get: ' + attr) # Not on attr1: inherited from class
if attr == 'attr3': # Not on attr2: stored on instance
return 3
else:
raise AttributeError(attr)
X = GetAttr()
print(X.attr1)
print(X.attr2)
print(X.attr3)
print('-'*20)
class GetAttribute(object): # (object) needed in 2.X only
attr1 = 1
def __init__(self):
self.attr2 = 2
def __getattribute__(self, attr): # On all attr fetches
print('get: ' + attr) # Use superclass to avoid looping here
if attr == 'attr3':
return 3
else:
return object.__getattribute__(self, attr)
X = GetAttribute()
print(X.attr1)
print(X.attr2)
print(X.attr3)c:\code> py −3 getattr-v-getattr.py
1
2
get: attr3
3
--------------------
get: attr1
1
get: attr2
2
get: attr3
3# Two dynamically computed attributes with propertiesclass Powers(object):# Need (object) in 2.X onlydef __init__(self, square, cube): self._square = square# _square is the base valueself._cube = cube# square is the property namedef getSquare(self): return self._square ** 2 def setSquare(self, value): self._square = value square = property(getSquare, setSquare) def getCube(self): return self._cube ** 3 cube = property(getCube) X = Powers(3, 4) print(X.square)# 3 ** 2 = 9print(X.cube)# 4 ** 3 = 64X.square = 5 print(X.square)# 5 ** 2 = 25
# Same, but with descriptors (per-instance state)class DescSquare(object): def __get__(self, instance, owner): return instance._square ** 2 def __set__(self, instance, value): instance._square = value class DescCube(object): def __get__(self, instance, owner): return instance._cube ** 3 class Powers(object):# Need all (object) in 2.X onlysquare = DescSquare() cube = DescCube() def __init__(self, square, cube): self._square = square# "self.square = square" works too,self._cube = cube# because it triggers desc __set__!X = Powers(3, 4) print(X.square)# 3 ** 2 = 9print(X.cube)# 4 ** 3 = 64X.square = 5 print(X.square)# 5 ** 2 = 25
# Same, but with generic __getattr__ undefined attribute interceptionclass Powers: def __init__(self, square, cube): self._square = square self._cube = cube def __getattr__(self, name): if name == 'square': return self._square ** 2 elif name == 'cube': return self._cube ** 3 else: raise TypeError('unknown attr:' + name) def __setattr__(self, name, value): if name == 'square': self.__dict__['_square'] = value# Or use objectelse: self.__dict__[name] = value X = Powers(3, 4) print(X.square)# 3 ** 2 = 9print(X.cube)# 4 ** 3 = 64X.square = 5 print(X.square)# 5 ** 2 = 25
# Same, but with generic __getattribute__ all attribute interceptionclass Powers(object):# Need (object) in 2.X onlydef __init__(self, square, cube): self._square = square self._cube = cube def __getattribute__(self, name): if name == 'square': return object.__getattribute__(self, '_square') ** 2 elif name == 'cube': return object.__getattribute__(self, '_cube') ** 3 else: return object.__getattribute__(self, name) def __setattr__(self, name, value): if name == 'square': object.__setattr__(self, '_square', value)# Or use __dict__else: object.__setattr__(self, name , value) X = Powers(3, 4) print(X.square)# 3 ** 2 = 9print(X.cube)# 4 ** 3 = 64X.square = 5 print(X.square)# 5 ** 2 = 25
9 64 25
__getattr__ nor __getattribute__ is run for such
attributes.__getattr__ is run
for such attributes if they are undefined in the class.__getattribute__ is available for
new-style classes only and works as it does in 3.X.class GetAttr:
eggs = 88 # eggs stored on class, spam on instance
def __init__(self):
self.spam = 77
def __len__(self): # len here, else __getattr__ called with __len__
print('__len__: 42')
return 42
def __getattr__(self, attr): # Provide __str__ if asked, else dummy func
print('getattr: ' + attr)
if attr == '__str__':
return lambda *args: '[Getattr str]'
else:
return lambda *args: None
class GetAttribute(object): # object required in 2.X, implied in 3.X
eggs = 88 # In 2.X all are isinstance(object) auto
def __init__(self): # But must derive to get new-style tools,
self.spam = 77 # incl __getattribute__, some __X__ defaults
def __len__(self):
print('__len__: 42')
return 42
def __getattribute__(self, attr):
print('getattribute: ' + attr)
if attr == '__str__':
return lambda *args: '[GetAttribute str]'
else:
return lambda *args: None
for Class in GetAttr, GetAttribute:
print('\n' + Class.__name__.ljust(50, '='))
X = Class()
X.eggs # Class attr
X.spam # Instance attr
X.other # Missing attr
len(X) # __len__ defined explicitly
# New-styles must support [], +, call directly: redefine
try: X[0] # __getitem__?
except: print('fail []')
try: X + 99 # __add__?
except: print('fail +')
try: X() # __call__? (implicit via built-in)
except: print('fail ()')
X.__call__() # __call__? (explicit, not inherited)
print(X.__str__()) # __str__? (explicit, inherited from type)
print(X) # __str__? (implicit via built-in)c:\code> py −2 getattr-builtins.py
GetAttr===========================================
getattr: other
__len__: 42
getattr: __getitem__
getattr: __coerce__
getattr: __add__
getattr: __call__
getattr: __call__
getattr: __str__
[Getattr str]
getattr: __str__
[Getattr str]
GetAttribute======================================
getattribute: eggs
getattribute: spam
getattribute: other
__len__: 42
fail []
fail +
fail ()
getattribute: __call__
getattribute: __str__
[GetAttribute str]
<__main__.GetAttribute object at 0x02287898> c:\code> py −3 getattr-builtins.py
GetAttr===========================================
getattr: other
__len__: 42
fail []
fail +
fail ()
getattr: __call__
<__main__.GetAttr object at 0x02987CC0>
<__main__.GetAttr object at 0x02987CC0>
GetAttribute======================================
getattribute: eggs
getattribute: spam
getattribute: other
__len__: 42
fail []
fail +
fail ()
getattribute: __call__
getattribute: __str__
[GetAttribute str]
<__main__.GetAttribute object at 0x02987CF8>__str__ access fails to be
caught twice by __getattr__ in
3.X: once for the built-in print, and once for explicit fetches
because a default is inherited from the class (really, from the
built-in object, which is an
automatic superclass to every class in 3.X).__str__ fails to be caught
only once by the __getattribute__
catchall, during the built-in print operation; explicit fetches
bypass the inherited version.__call__ fails to be caught
in both schemes in 3.X for built-in call expressions, but it is
intercepted by both when fetched explicitly; unlike __str__, there is no inherited __call__ default for object instances to defeat __getattr__.__len__ is caught by both
classes, simply because it is an explicitly defined method in the
classes themselves — though its name it is not routed to either
__getattr__ or __getattribute__ in 3.X if we delete the
class’s __len__ methods.class Person:
def __init__(self, name, job=None, pay=0):
self.name = name
self.job = job
self.pay = pay
def lastName(self):
return self.name.split()[-1]
def giveRaise(self, percent):
self.pay = int(self.pay * (1 + percent))
def __repr__(self):
return '[Person: %s, %s]' % (self.name, self.pay)
class Manager:
def __init__(self, name, pay):
self.person = Person(name, 'mgr', pay) # Embed a Person object
def giveRaise(self, percent, bonus=.10):
self.person.giveRaise(percent + bonus) # Intercept and delegate
def __getattr__(self, attr):
return getattr(self.person, attr) # Delegate all other attrs
def __repr__(self):
return str(self.person) # Must overload again (in 3.X)
if __name__ == '__main__':
sue = Person('Sue Jones', job='dev', pay=100000)
print(sue.lastName())
sue.giveRaise(.10)
print(sue)
tom = Manager('Tom Jones', 50000) # Manager.__init__
print(tom.lastName()) # Manager.__getattr__ -> Person.lastName
tom.giveRaise(.10) # Manager.giveRaise -> Person.giveRaise
print(tom) # Manager.__repr__ -> Person.__repr__c:\code> py −3 getattr-delegate.py
Jones
[Person: Sue Jones, 110000]
Jones
[Person: Tom Jones, 60000]# Delete the Manager __str__ methodclass Manager: def __init__(self, name, pay): self.person = Person(name, 'mgr', pay)# Embed a Person objectdef giveRaise(self, percent, bonus=.10): self.person.giveRaise(percent + bonus)# Intercept and delegatedef __getattr__(self, attr): return getattr(self.person, attr)# Delegate all other attrs
c:\code> py −3 getattr-delegate.py
Jones
[Person: Sue Jones, 110000]
Jones
<__main__.Manager object at 0x029E7B70>c:\code> py −2 getattr-delegate.py
Jones
[Person: Sue Jones, 110000]
Jones
[Person: Tom Jones, 60000]# Replace __getattr_ with __getattribute__class Manager(object):# Use "(object)" in 2.Xdef __init__(self, name, pay): self.person = Person(name, 'mgr', pay)# Embed a Person objectdef giveRaise(self, percent, bonus=.10): self.person.giveRaise(percent + bonus)# Intercept and delegatedef __getattribute__(self, attr): print('**', attr) if attr in ['person', 'giveRaise']: return object.__getattribute__(self, attr)# Fetch my attrselse: return getattr(self.person, attr)# Delegate all others
C:\code> py −3 getattr-delegate.py
Jones
[Person: Sue Jones, 110000]
** lastName
** person
Jones
** giveRaise
** person
<__main__.Manager object at 0x028E0590># Code __getattribute__ differently to minimize extra calls
class Manager:
def __init__(self, name, pay):
self.person = Person(name, 'mgr', pay)
def __getattribute__(self, attr):
print('**', attr)
person = object.__getattribute__(self, 'person')
if attr == 'giveRaise':
return lambda percent: person.giveRaise(percent+.10)
else:
return getattr(person, attr)
def __repr__(self):
person = object.__getattribute__(self, 'person')
return str(person)Jones [Person: Sue Jones, 110000] ** lastName Jones ** giveRaise [Person: Tom Jones, 60000]
# File validate_properties.pyclass CardHolder(object):# Need "(object)" for setter in 2.Xacctlen = 8# Class dataretireage = 59.5 def __init__(self, acct, name, age, addr): self.acct = acct# Instance dataself.name = name# These trigger prop setters too!self.age = age# __X mangled to have class nameself.addr = addr# addr is not managed# remain has no datadef getName(self): return self.__name def setName(self, value): value = value.lower().replace(' ', '_') self.__name = value name = property(getName, setName) def getAge(self): return self.__age def setAge(self, value): if value < 0 or value > 150: raise ValueError('invalid age') else: self.__age = value age = property(getAge, setAge) def getAcct(self): return self.__acct[:-3] + '***' def setAcct(self, value): value = value.replace('-', '') if len(value) != self.acctlen: raise TypeError('invald acct number') else: self.__acct = value acct = property(getAcct, setAcct) def remainGet(self):# Could be a method, not attrreturn self.retireage - self.age# Unless already using as attrremain = property(remainGet)
# File validate_tester.pyfrom __future__ import print_function# 2.Xdef loadclass(): import sys, importlib modulename = sys.argv[1]# Module name in command linemodule = importlib.import_module(modulename)# Import module by name stringprint('[Using: %s]' % module.CardHolder)# No need for getattr() herereturn module.CardHolder def printholder(who): print(who.acct, who.name, who.age, who.remain, who.addr, sep=' / ') if __name__ == '__main__': CardHolder = loadclass() bob = CardHolder('1234-5678', 'Bob Smith', 40, '123 main st') printholder(bob) bob.name = 'Bob Q. Smith' bob.age = 50 bob.acct = '23-45-67-89' printholder(bob) sue = CardHolder('5678-12-34', 'Sue Jones', 35, '124 main st') printholder(sue) try: sue.age = 200 except: print('Bad age for Sue') try: sue.remain = 5 except: print("Can't set sue.remain") try: sue.acct = '1234567' except: print('Bad acct for Sue')
c:\code> py −3 validate_tester.py validate_properties
[Using: <class 'validate_properties.CardHolder'>]
12345*** / bob_smith / 40 / 19.5 / 123 main st
23456*** / bob_q._smith / 50 / 9.5 / 123 main st
56781*** / sue_jones / 35 / 24.5 / 124 main st
Bad age for Sue
Can't set sue.remain
Bad acct for Sue# File validate_tester2.pyfrom __future__ import print_function# 2.Xfrom validate_tester import loadclass CardHolder = loadclass() bob = CardHolder('1234-5678', 'Bob Smith', 40, '123 main st') print('bob:', bob.name, bob.acct, bob.age, bob.addr) sue = CardHolder('5678-12-34', 'Sue Jones', 35, '124 main st') print('sue:', sue.name, sue.acct, sue.age, sue.addr)# addr differs: client dataprint('bob:', bob.name, bob.acct, bob.age, bob.addr)# name,acct,age overwritten?
c:\code> py −3 validate_tester2.py validate_descriptors1
[Using: <class 'validate_descriptors1.CardHolder'>]
bob: bob_smith 12345*** 40 123 main st
sue: sue_jones 56781*** 35 124 main st
bob: sue_jones 56781*** 35 123 main st# File validate_descriptors2.py: using per-client-instance stateclass CardHolder(object):# Need all "(object)" in 2.X onlyacctlen = 8# Class dataretireage = 59.5 def __init__(self, acct, name, age, addr): self.acct = acct# Client instance dataself.name = name# These trigger __set__ calls too!self.age = age #__X needed: in client instanceself.addr = addr# addr is not managed# remain managed but has no dataclass Name(object): def __get__(self, instance, owner):# Class names: CardHolder localsreturn instance.__name def __set__(self, instance, value): value = value.lower().replace(' ', '_') instance.__name = value name = Name()# class.name vs mangled attrclass Age(object): def __get__(self, instance, owner): return instance.__age# Use descriptor datadef __set__(self, instance, value): if value < 0 or value > 150: raise ValueError('invalid age') else: instance.__age = value age = Age()# class.age vs mangled attrclass Acct(object): def __get__(self, instance, owner): return instance.__acct[:-3] + '***' def __set__(self, instance, value): value = value.replace('-', '') if len(value) != instance.acctlen:# Use instance class dataraise TypeError('invald acct number') else: instance.__acct = value acct = Acct()# class.acct vs mangled nameclass Remain(object): def __get__(self, instance, owner): return instance.retireage - instance.age# Triggers Age.__get__def __set__(self, instance, value): raise TypeError('cannot set remain')# Else set allowed hereremain = Remain()
c:\code>py −3 validate_tester2.py validate_descriptors2[Using: <class 'validate_descriptors2.CardHolder'>] bob: bob_smith 12345*** 40 123 main st sue: sue_jones 56781*** 35 124 main st bob: bob_smith 12345*** 40 123 main st c:\code>py −3 validate_tester.py validate_descriptors2...same output as properties, except class name...
>>>from validate_descriptors1 import CardHolder>>>bob = CardHolder('1234-5678', 'Bob Smith', 40, '123 main st')>>>bob.name'bob_smith' >>>CardHolder.name'bob_smith' >>>from validate_descriptors2 import CardHolder>>>bob = CardHolder('1234-5678', 'Bob Smith', 40, '123 main st')>>>bob.name'bob_smith' >>>CardHolder.nameAttributeError: 'NoneType' object has no attribute '_Name__name'
# File validate_getattr.pyclass CardHolder: acctlen = 8# Class dataretireage = 59.5 def __init__(self, acct, name, age, addr): self.acct = acct# Instance dataself.name = name# These trigger __setattr__ tooself.age = age# _acct not mangled: name testedself.addr = addr# addr is not managed# remain has no datadef __getattr__(self, name): if name == 'acct':# On undefined attr fetchesreturn self._acct[:-3] + '***'# name, age, addr are definedelif name == 'remain': return self.retireage - self.age# Doesn't trigger __getattr__else: raise AttributeError(name) def __setattr__(self, name, value): if name == 'name':# On all attr assignmentsvalue = value.lower().replace(' ', '_')# addr stored directlyelif name == 'age':# acct mangled to _acctif value < 0 or value > 150: raise ValueError('invalid age') elif name == 'acct': name = '_acct' value = value.replace('-', '') if len(value) != self.acctlen: raise TypeError('invald acct number') elif name == 'remain': raise TypeError('cannot set remain') self.__dict__[name] = value# Avoid looping (or via object)
c:\code>py −3 validate_tester.py validate_getattr...same output as properties, except class name...c:\code>py −3 validate_tester2.py validate_getattr...same output as instance-state descriptors, except class name...
# File validate_getattribute.pyclass CardHolder(object):# Need "(object)" in 2.X onlyacctlen = 8# Class dataretireage = 59.5 def __init__(self, acct, name, age, addr): self.acct = acct# Instance dataself.name = name# These trigger __setattr__ tooself.age = age# acct not mangled: name testedself.addr = addr# addr is not managed# remain has no datadef __getattribute__(self, name): superget = object.__getattribute__# Don't loop: one level upif name == 'acct':# On all attr fetchesreturn superget(self, 'acct')[:-3] + '***' elif name == 'remain': return superget(self, 'retireage') - superget(self, 'age') else: return superget(self, name)# name, age, addr: storeddef __setattr__(self, name, value): if name == 'name':# On all attr assignmentsvalue = value.lower().replace(' ', '_')# addr stored directlyelif name == 'age': if value < 0 or value > 150: raise ValueError('invalid age') elif name == 'acct': value = value.replace('-', '') if len(value) != self.acctlen: raise TypeError('invald acct number') elif name == 'remain': raise TypeError('cannot set remain') self.__dict__[name] = value# Avoid loops, orig names
__getattr__ and
__getattribute__ differ?__getattr__ and __getattribute__ and properties and
descriptors?__getattr__ method is
run for fetches of undefined attributes only
(i.e., those not present on an instance and not inherited from any
of its classes). By contrast, the __getattribute__ method is called for
every attribute fetch, whether the attribute is
defined or not. Because of this, code inside a __getattr__ can freely fetch other
attributes if they are defined, whereas __getattribute__ must use special code for
all such attribute fetches to avoid looping or extra calls (it must
route fetches to a superclass to skip itself).property built-in accepts a
single function argument, it can be used directly as a function
decorator to define a fetch access property. Due to the name
rebinding behavior of decorators, the name of the decorated function
is assigned to a property whose get accessor is set to the original
function decorated (name =
property(name)). Property setter and deleter attributes allow us to further add
set and delete accessors with decoration syntax — they set the
accessor to the decorated function and return the augmented
property.__getattr__ and
__getattribute__ methods are more
generic: they can be used to catch arbitrarily many attributes. In
contrast, each property or descriptor provides access interception
for only one specific attribute — we can’t catch
every attribute fetch with a single property or descriptor. On the
other hand, properties and descriptors handle both attribute fetch
and assignment by design: __getattr__ and __getattribute__ handle fetches only; to
intercept assignments as well, __setattr__ must also be coded. The
implementation is also different: __getattr__ and __getattribute__ are operator overloading
methods, whereas properties and descriptors are objects manually
assigned to class attributes. Unlike the others, properties and
descriptors can also sometimes avoid extra calls on assignment to
unmanaged names, and show up in dir results automatically, but are also
narrower in scope — they can’t address generic dispatch goals. In
Python evolution, new features tend to offer alternatives, but do
not fully subsume what came before.An argument is a connected series of statements intended to establish a
proposition.
No it isn't.
Yes it is! It's not just contradiction.
Look, if I argue with you, I must take up a contrary position.
Yes, but that's not just saying "No it isn't."
Yes it is!
No it isn't!
Yes it is!
No it isn't. Argument is an intellectual process. Contradiction is just
the automatic gainsaying of any statement the other person makes.
(short pause) No it isn't.
It is.
Not at all.
Now look...1 As noted in Chapter 31, such
dynamic classes can also use a __dir__ method to
provide an attribute result list for dir calls,
though general tools cannot depend on this optional
interface.
Function decorators install wrapper objects to intercept later function calls and process them as needed, usually passing the call on to the original function to run the managed action.
Class decorators install wrapper objects to intercept later instance creation calls and process them as required, usually passing the call on to the original class to create a managed instance.
Function decorators can also be used to manage function objects, instead of or in addition to later calls to them — to register a function to an API, for instance. Our primary focus here, though, will be on their more commonly used call wrapper application.
Class decorators can also be used to manage class objects directly, instead of or in addition to instance creation calls — to augment a class with new methods, for example. Because this role intersects strongly with that of metaclasses, we’ll see additional use cases in the next chapter. As we’ll find, both tools run at the end of the class creation process, but class decorators often offer a lighter-weight solution.
@decorator# Decorate functiondef F(arg): ... F(99)# Call function
def F(arg):
...
F = decorator(F) # Rebind function name to decorator result
F(99) # Essentially calls decorator(F)(99)func(6, 7) decorator(func)(6, 7)
class C:
@staticmethod
def meth(...): ... # meth = staticmethod(meth)
class C:
@property
def name(self): ... # name = property(name)def decorator(F):
# Process function F
return F
@decorator
def func(): ... # func = decorator(func)def decorator(F):
# Save or use function F
# Return a different callable: nested def, class with __call__, etc.
@decorator
def func(): ... # func = decorator(func)def decorator(F):# On @ decorationdef wrapper(*args):# On wrapped function call# Use F and args# F(*args) calls original functionreturn wrapper @decorator# func = decorator(func)def func(x, y):# func is passed to decorator's F... func(6, 7)# 6, 7 are passed to wrapper's *args
class decorator:
def __init__(self, func): # On @ decoration
self.func = func
def __call__(self, *args): # On wrapped function call
# Use self.func and args
# self.func(*args) calls original function
@decorator
def func(x, y): # func = decorator(func)
... # func is passed to __init__
func(6, 7) # 6, 7 are passed to __call__'s *argsclass decorator:
def __init__(self, func): # func is method without instance
self.func = func
def __call__(self, *args): # self is decorator instance
# self.func(*args) fails! # C instance not in args!
class C:
@decorator
def method(self, x, y): # method = decorator(method)
... # Rebound to decorator instancedef decorator(F):# F is func or method without instancedef wrapper(*args):# class instance in args[0] for method# F(*args) runs func or methodreturn wrapper @decorator def func(x, y):# func = decorator(func)... func(6, 7)# Really calls wrapper(6, 7)class C: @decorator def method(self, x, y):# method = decorator(method)...# Rebound to simple functionX = C() X.method(6, 7)# Really calls wrapper(X, 6, 7)
@decorator# Decorate classclass C: ... x = C(99)# Make an instance
class C:
...
C = decorator(C) # Rebind class name to decorator result
x = C(99) # Essentially calls decorator(C)(99)def decorator(C):
# Process class C
return C
@decorator
class C: ... # C = decorator(C)def decorator(C):
# Save or use class C
# Return a different callable: nested def, class with __call__, etc.
@decorator
class C: ... # C = decorator(C)def decorator(cls):# On @ decorationclass Wrapper: def __init__(self, *args):# On instance creationself.wrapped = cls(*args) def __getattr__(self, name):# On attribute fetchreturn getattr(self.wrapped, name) return Wrapper @decorator class C:# C = decorator(C)def __init__(self, x, y):# Run by Wrapper.__init__self.attr = 'spam' x = C(6, 7)# Really calls Wrapper(6, 7)print(x.attr)# Runs Wrapper.__getattr__, prints "spam"
class Decorator:
def __init__(self, C): # On @ decoration
self.C = C
def __call__(self, *args): # On instance creation
self.wrapped = self.C(*args)
return self
def __getattr__(self, attrname): # On atrribute fetch
return getattr(self.wrapped, attrname)
@Decorator
class C: ... # C = Decorator(C)
x = C()
y = C() # Overwrites x!def decorator(C):# On @ decorationclass Wrapper: def __init__(self, *args):# On instance creation: new Wrapperself.wrapped = C(*args)# Embed instance in instancereturn Wrapper class Wrapper: ... def decorator(C):# On @ decorationdef onCall(*args):# On instance creation: new Wrapperreturn Wrapper(C(*args))# Embed instance in instancereturn onCall
@A
@B
@C
def f(...):
...def f(...):
...
f = A(B(C(f)))@spam
@eggs
class C:
...
X = C()class C:
...
C = spam(eggs(C))
X = C()def d1(F): return F def d2(F): return F def d3(F): return F @d1 @d2 @d3 def func():# func = d1(d2(d3(func)))print('spam') func()# Prints "spam"
def d1(F): return lambda: 'X' + F() def d2(F): return lambda: 'Y' + F() def d3(F): return lambda: 'Z' + F() @d1 @d2 @d3 def func():# func = d1(d2(d3(func)))return 'spam' print(func())# Prints "XYZspam"
@decorator(A, B)
def F(arg):
...
F(99)def F(arg):
...
F = decorator(A, B)(F) # Rebind F to result of decorator's return value
F(99) # Essentially calls decorator(A, B)(F)(99)def decorator(A, B):
# Save or use A, B
def actualDecorator(F):
# Save or use function F
# Return a callable: nested def, class with __call__, etc.
return callable
return actualDecoratordef decorator(O):
# Save or augment function or class O
return O
@decorator
def F(): ... # F = decorator(F)
@decorator
class C: ... # C = decorator(C)# File decorator1.pyclass tracer: def __init__(self, func):# On @ decoration: save original funcself.calls = 0 self.func = func def __call__(self, *args):# On later calls: run original funcself.calls += 1 print('call %s to %s' % (self.calls, self.func.__name__)) self.func(*args) @tracer def spam(a, b, c):# spam = tracer(spam)print(a + b + c)# Wraps spam in a decorator object
>>>from decorator1 import spam>>>spam(1, 2, 3)# Really calls the tracer wrapper objectcall 1 to spam 6 >>>spam('a', 'b', 'c')# Invokes __call__ in classcall 2 to spam abc >>>spam.calls# Number calls in wrapper state information2 >>>spam<decorator1.tracer object at 0x02D9A730>
calls = 0
def tracer(func, *args):
global calls
calls += 1
print('call %s to %s' % (calls, func.__name__))
func(*args)
def spam(a, b, c):
print(a, b, c)
>>> spam(1, 2, 3) # Normal nontraced call: accidental?
1 2 3
>>> tracer(spam, 1, 2, 3) # Special traced call without decorators
call 1 to spam
1 2 3class tracer:# State via instance attributesdef __init__(self, func):# On @ decoratorself.calls = 0# Save func for later callself.func = func def __call__(self, *args, **kwargs):# On call to original functionself.calls += 1 print('call %s to %s' % (self.calls, self.func.__name__)) return self.func(*args, **kwargs) @tracer def spam(a, b, c):# Same as: spam = tracer(spam)print(a + b + c)# Triggers tracer.__init__@tracer def eggs(x, y):# Same as: eggs = tracer(eggs)print(x ** y)# Wraps eggs in a tracer objectspam(1, 2, 3)# Really calls tracer instance: runs tracer.__call__spam(a=4, b=5, c=6)# spam is an instance attributeeggs(2, 16)# Really calls tracer instance, self.func is eggseggs(4, y=4)# self.calls is per-decoration here
c:\code> python decorator2.py
call 1 to spam
6
call 2 to spam
15
call 1 to eggs
65536
call 2 to eggs
256calls = 0 def tracer(func):# State via enclosing scope and globaldef wrapper(*args, **kwargs):# Instead of class attributesglobal calls# calls is global, not per-functioncalls += 1 print('call %s to %s' % (calls, func.__name__)) return func(*args, **kwargs) return wrapper @tracer def spam(a, b, c):# Same as: spam = tracer(spam)print(a + b + c) @tracer def eggs(x, y):# Same as: eggs = tracer(eggs)print(x ** y) spam(1, 2, 3)# Really calls wrapper, assigned to spamspam(a=4, b=5, c=6)# wrapper calls spameggs(2, 16)# Really calls wrapper, assigned to eggseggs(4, y=4)# Global calls is not per-decoration here!
c:\code> python decorator3.py
call 1 to spam
6
call 2 to spam
15
call 3 to eggs
65536
call 4 to eggs
256def tracer(func):# State via enclosing scope and nonlocalcalls = 0# Instead of class attrs or globaldef wrapper(*args, **kwargs):# calls is per-function, not globalnonlocal calls calls += 1 print('call %s to %s' % (calls, func.__name__)) return func(*args, **kwargs) return wrapper @tracer def spam(a, b, c):# Same as: spam = tracer(spam)print(a + b + c) @tracer def eggs(x, y):# Same as: eggs = tracer(eggs)print(x ** y) spam(1, 2, 3)# Really calls wrapper, bound to funcspam(a=4, b=5, c=6)# wrapper calls spameggs(2, 16)# Really calls wrapper, bound to eggseggs(4, y=4)# Nonlocal calls _is_ per-decoration here
c:\code> py −3 decorator4.py
call 1 to spam
6
call 2 to spam
15
call 1 to eggs
65536
call 2 to eggs
256def tracer(func):# State via enclosing scope and func attrdef wrapper(*args, **kwargs):# calls is per-function, not globalwrapper.calls += 1 print('call %s to %s' % (wrapper.calls, func.__name__)) return func(*args, **kwargs) wrapper.calls = 0 return wrapper @tracer def spam(a, b, c):# Same as: spam = tracer(spam)print(a + b + c) @tracer def eggs(x, y):# Same as: eggs = tracer(eggs)print(x ** y) spam(1, 2, 3)# Really calls wrapper, assigned to spamspam(a=4, b=5, c=6)# wrapper calls spameggs(2, 16)# Really calls wrapper, assigned to eggseggs(4, y=4)# wrapper.calls _is_ per-decoration here
c:\code>py −2 decorator5.py...same output as prior version, but works on 2.X too...
class tracer:
def __init__(self, func): # On @ decorator
self.calls = 0 # Save func for later call
self.func = func
def __call__(self, *args, **kwargs): # On call to original function
self.calls += 1
print('call %s to %s' % (self.calls, self.func.__name__))
return self.func(*args, **kwargs)@tracer def spam(a, b, c):# spam = tracer(spam)print(a + b + c)# Triggers tracer.__init__>>>spam(1, 2, 3)# Runs tracer.__call__call 1 to spam 6 >>>spam(a=4, b=5, c=6)# spam saved in an instance attributecall 2 to spam 15
class Person:
def __init__(self, name, pay):
self.name = name
self.pay = pay
@tracer
def giveRaise(self, percent): # giveRaise = tracer(giveRaise)
self.pay *= (1.0 + percent)
@tracer
def lastName(self): # lastName = tracer(lastName)
return self.name.split()[-1]
>>> bob = Person('Bob Smith', 50000) # tracer remembers method funcs
>>> bob.giveRaise(.25) # Runs tracer.__call__(???, .25)
call 1 to giveRaise
TypeError: giveRaise() missing 1 required positional argument: 'percent'
>>> print(bob.lastName()) # Runs tracer.__call__(???)
call 1 to lastName
TypeError: lastName() missing 1 required positional argument: 'self'>>> bob.giveRaise(.25)
<__main__.tracer object at 0x02A486D8> (0.25,) {}
call 1 to giveRaise
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 9, in __call__
TypeError: giveRaise() missing 1 required positional argument: 'percent'# A call tracer decorator for both functions and methodsdef tracer(func):# Use function, not class with __call__calls = 0# Else "self" is decorator instance only!def onCall(*args, **kwargs):# Or in 2.X+3.X: use [onCall.calls += 1]nonlocal calls calls += 1 print('call %s to %s' % (calls, func.__name__)) return func(*args, **kwargs) return onCall if __name__ == '__main__':# Applies to simple functions@tracer def spam(a, b, c):# spam = tracer(spam)print(a + b + c)# onCall remembers spam@tracer def eggs(N): return 2 ** N spam(1, 2, 3)# Runs onCall(1, 2, 3)spam(a=4, b=5, c=6) print(eggs(32))# Applies to class-level method functions too!class Person: def __init__(self, name, pay): self.name = name self.pay = pay @tracer def giveRaise(self, percent):# giveRaise = tracer(giveRaise)self.pay *= (1.0 + percent)# onCall remembers giveRaise@tracer def lastName(self):# lastName = tracer(lastName)return self.name.split()[-1] print('methods...') bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) print(bob.name, sue.name) sue.giveRaise(.10)# Runs onCall(sue, .10)print(int(sue.pay)) print(bob.lastName(), sue.lastName())# Runs onCall(bob), lastName in scopes
c:\code> py −3 calltracer.py
call 1 to spam
6
call 2 to spam
15
call 1 to eggs
4294967296
methods...
Bob Smith Sue Jones
call 1 to giveRaise
110000
call 1 to lastName
call 2 to lastName
Smith Jonesclass Descriptor(object):
def __get__(self, instance, owner): ...
class Subject:
attr = Descriptor()
X = Subject()
X.attr # Roughly runs Descriptor.__get__(Subject.attr, X, Subject)class tracer(object):# A decorator+descriptordef __init__(self, func):# On @ decoratorself.calls = 0# Save func for later callself.func = func def __call__(self, *args, **kwargs):# On call to original funcself.calls += 1 print('call %s to %s' % (self.calls, self.func.__name__)) return self.func(*args, **kwargs) def __get__(self, instance, owner):# On method attribute fetchreturn wrapper(self, instance) class wrapper: def __init__(self, desc, subj):# Save both instancesself.desc = desc# Route calls back to deco/descself.subj = subj def __call__(self, *args, **kwargs): return self.desc(self.subj, *args, **kwargs)# Runs tracer.__call__@tracer def spam(a, b, c):# spam = tracer(spam)...same as prior...# Uses __call__ onlyclass Person: @tracer def giveRaise(self, percent):# giveRaise = tracer(giveRaise)...same as prior...# Makes giveRaise a descriptor
__call__, and never invoke its __get__.__get__ first to resolve the method name
fetch (on I.method); the
object returned by __get__
retains the subject class instance and is then invoked to complete
the call expression, thereby triggering the decorator’s __call__ (on ()).sue.giveRaise(.10) # Runs __get__ then __call__class tracer(object):
def __init__(self, func): # On @ decorator
self.calls = 0 # Save func for later call
self.func = func
def __call__(self, *args, **kwargs): # On call to original func
self.calls += 1
print('call %s to %s' % (self.calls, self.func.__name__))
return self.func(*args, **kwargs)
def __get__(self, instance, owner): # On method fetch
def wrapper(*args, **kwargs): # Retain both inst
return self(instance, *args, **kwargs) # Runs __call__
return wrapperclass tracer(object):# For methods, but not functions!def __init__(self, meth):# On @ decoratorself.calls = 0 self.meth = meth def __get__(self, instance, owner):# On method fetchdef wrapper(*args, **kwargs):# On method call: proxy with self+instself.calls += 1 print('call %s to %s' % (self.calls, self.meth.__name__)) return self.meth(instance, *args, **kwargs) return wrapper class Person: @tracer# Applies to class methodsdef giveRaise(self, percent):# giveRaise = tracer(giveRaise)...# Makes giveRaise a descriptor@tracer# But fails for simple functionsdef spam(a, b, c):# spam = tracer(spam)...# No attribute fetch occurs here
# File timerdeco1.py# Caveat: range still differs - a list in 2.X, an iterable in 3.X# Caveat: timer won't work on methods as coded (see quiz solution)import time, sys force = list if sys.version_info[0] == 3 else (lambda X: X) class timer: def __init__(self, func): self.func = func self.alltime = 0 def __call__(self, *args, **kargs): start = time.clock() result = self.func(*args, **kargs) elapsed = time.clock() - start self.alltime += elapsed print('%s: %.5f, %.5f' % (self.func.__name__, elapsed, self.alltime)) return result @timer def listcomp(N): return [x * 2 for x in range(N)] @timer def mapcall(N): return force(map((lambda x: x * 2), range(N))) result = listcomp(5)# Time for this call, all calls, return valuelistcomp(50000) listcomp(500000) listcomp(1000000) print(result) print('allTime = %s' % listcomp.alltime)# Total time for all listcomp callsprint('') result = mapcall(5) mapcall(50000) mapcall(500000) mapcall(1000000) print(result) print('allTime = %s' % mapcall.alltime)# Total time for all mapcall callsprint('\n**map/comp = %s' % round(mapcall.alltime / listcomp.alltime, 3))
c:\code> py −3 timerdeco1.py
listcomp: 0.00001, 0.00001
listcomp: 0.00499, 0.00499
listcomp: 0.05716, 0.06215
listcomp: 0.11565, 0.17781
[0, 2, 4, 6, 8]
allTime = 0.17780527629411225
mapcall: 0.00002, 0.00002
mapcall: 0.00988, 0.00990
mapcall: 0.10601, 0.11591
mapcall: 0.21690, 0.33281
[0, 2, 4, 6, 8]
allTime = 0.3328064956447921
**map/comp = 1.872>>>def listcomp(N): [x * 2 for x in range(N)]>>>import timer# Chapter 21 techniques>>>timer.total(1, listcomp, 1000000)(0.1461295268088542, None) >>>import timeit>>>timeit.timeit(number=1, stmt=lambda: listcomp(1000000))0.14964829430189397
def timer(label=''):
def decorator(func):
def onCall(*args): # Multilevel state retention:
... # args passed to function
func(*args) # func retained in enclosing scope
print(label, ... # label retained in enclosing scope
return onCall
return decorator # Returns the actual decorator
@timer('==>') # Like listcomp = timer('==>')(listcomp)
def listcomp(N): ... # listcomp is rebound to new onCall
listcomp(...) # Really calls onCallimport time def timer(label='', trace=True):# On decorator args: retain argsclass Timer: def __init__(self, func):# On @: retain decorated funcself.func = func self.alltime = 0 def __call__(self, *args, **kargs):# On calls: call originalstart = time.clock() result = self.func(*args, **kargs) elapsed = time.clock() - start self.alltime += elapsed if trace: format = '%s %s: %.5f, %.5f' values = (label, self.func.__name__, elapsed, self.alltime) print(format % values) return result return Timer
import sys from timerdeco2 import timer force = list if sys.version_info[0] == 3 else (lambda X: X) @timer(label='[CCC]==>') def listcomp(N):# Like listcomp = timer(...)(listcomp)return [x * 2 for x in range(N)]# listcomp(...) triggers Timer.__call__@timer(trace=True, label='[MMM]==>') def mapcall(N): return force(map((lambda x: x * 2), range(N))) for func in (listcomp, mapcall): result = func(5)# Time for this call, all calls, return valuefunc(50000) func(500000) func(1000000) print(result) print('allTime = %s\n' % func.alltime)# Total time for all callsprint('**map/comp = %s' % round(mapcall.alltime / listcomp.alltime, 3))
c:\code> py −3 testseqs.py
[CCC]==> listcomp: 0.00001, 0.00001
[CCC]==> listcomp: 0.00504, 0.00505
[CCC]==> listcomp: 0.05839, 0.06344
[CCC]==> listcomp: 0.12001, 0.18344
[0, 2, 4, 6, 8]
allTime = 0.1834406801777564
[MMM]==> mapcall: 0.00003, 0.00003
[MMM]==> mapcall: 0.00961, 0.00964
[MMM]==> mapcall: 0.10929, 0.11892
[MMM]==> mapcall: 0.22143, 0.34035
[0, 2, 4, 6, 8]
allTime = 0.3403542519173618
**map/comp = 1.855>>>from timerdeco2 import timer>>>@timer(trace=False)# No tracing, collect total time...def listcomp(N):...return [x * 2 for x in range(N)]... >>>x = listcomp(5000)>>>x = listcomp(5000)>>>x = listcomp(5000)>>>listcomp.alltime0.0037191417530599152 >>>listcomp<timerdeco2.timer.<locals>.Timer object at 0x02957518> >>>@timer(trace=True, label='\t=>')# Turn on tracing, custom label...def listcomp(N):...return [x * 2 for x in range(N)]... >>>x = listcomp(5000)=> listcomp: 0.00106, 0.00106 >>>x = listcomp(5000)=> listcomp: 0.00108, 0.00214 >>>x = listcomp(5000)=> listcomp: 0.00107, 0.00321 >>>listcomp.alltime0.003208920466562404
# 3.X and 2.X: global tableinstances = {} def singleton(aClass):# On @ decorationdef onCall(*args, **kwargs):# On instance creationif aClass not in instances:# One dict entry per classinstances[aClass] = aClass(*args, **kwargs) return instances[aClass] return onCall
@singleton# Person = singleton(Person)class Person:# Rebinds Person to onCalldef __init__(self, name, hours, rate):# onCall remembers Personself.name = name self.hours = hours self.rate = rate def pay(self): return self.hours * self.rate @singleton# Spam = singleton(Spam)class Spam:# Rebinds Spam to onCalldef __init__(self, val):# onCall remembers Spamself.attr = val bob = Person('Bob', 40, 10)# Really calls onCallprint(bob.name, bob.pay()) sue = Person('Sue', 50, 20)# Same, single objectprint(sue.name, sue.pay()) X = Spam(val=42)# One Person, one SpamY = Spam(99) print(X.attr, Y.attr)
c:\code> python singletons.py
Bob 400
Bob 400
42 42# 3.X only: nonlocaldef singleton(aClass):# On @ decorationinstance = None def onCall(*args, **kwargs):# On instance creationnonlocal instance# 3.X and later nonlocalif instance == None: instance = aClass(*args, **kwargs)# One scope per classreturn instance return onCall
# 3.X and 2.X: func attrs, classes (alternative codings)def singleton(aClass):# On @ decorationdef onCall(*args, **kwargs):# On instance creationif onCall.instance == None: onCall.instance = aClass(*args, **kwargs)# One function per classreturn onCall.instance onCall.instance = None return onCall class singleton: def __init__(self, aClass):# On @ decorationself.aClass = aClass self.instance = None def __call__(self, *args, **kwargs):# On instance creationif self.instance == None: self.instance = self.aClass(*args, **kwargs)# One instance per classreturn self.instance
class Wrapper:
def __init__(self, object):
self.wrapped = object # Save object
def __getattr__(self, attrname):
print('Trace:', attrname) # Trace fetch
return getattr(self.wrapped, attrname) # Delegate fetch
>>> x = Wrapper([1,2,3]) # Wrap a list
>>> x.append(4) # Delegate to list method
Trace: append
>>> x.wrapped # Print my member
[1, 2, 3, 4]
>>> x = Wrapper({"a": 1, "b": 2}) # Wrap a dictionary
>>> list(x.keys()) # Delegate to dictionary method
Trace: keys # Use list() in 3.X
['a', 'b']def Tracer(aClass):# On @ decoratorclass Wrapper: def __init__(self, *args, **kargs):# On instance creationself.fetches = 0 self.wrapped = aClass(*args, **kargs)# Use enclosing scope namedef __getattr__(self, attrname): print('Trace: ' + attrname)# Catches all but own attrsself.fetches += 1 return getattr(self.wrapped, attrname)# Delegate to wrapped objreturn Wrapper if __name__ == '__main__': @Tracer class Spam:# Spam = Tracer(Spam)def display(self):# Spam is rebound to Wrapperprint('Spam!' * 8) @Tracer class Person:# Person = Tracer(Person)def __init__(self, name, hours, rate):# Wrapper remembers Personself.name = name self.hours = hours self.rate = rate def pay(self):# Accesses outside class tracedreturn self.hours * self.rate# In-method accesses not tracedfood = Spam()# Triggers Wrapper()food.display()# Triggers __getattr__print([food.fetches]) bob = Person('Bob', 40, 50)# bob is really a Wrapperprint(bob.name)# Wrapper embeds a Personprint(bob.pay()) print('') sue = Person('Sue', rate=100, hours=60)# sue is a different Wrapperprint(sue.name)# with a different Personprint(sue.pay()) print(bob.name)# bob has different stateprint(bob.pay()) print([bob.fetches, sue.fetches])# Wrapper attrs not traced
c:\code> python interfacetracer.py
Trace: display
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!
[1]
Trace: name
Bob
Trace: pay
2000
Trace: name
Sue
Trace: pay
6000
Trace: name
Bob
Trace: pay
2000
[4, 2]>>>from interfacetracer import Tracer>>>@Tracer...class MyList(list): pass# MyList = Tracer(MyList)>>>x = MyList([1, 2, 3])# Triggers Wrapper()>>>x.append(4)# Triggers __getattr__, appendTrace: append >>>x.wrapped[1, 2, 3, 4] >>>WrapList = Tracer(list)# Or perform decoration manually>>>x = WrapList([4, 5, 6])# Else subclass statement required>>>x.append(7)Trace: append >>>x.wrapped[4, 5, 6, 7]
@Tracer# Decorator approachclass Person: ... bob = Person('Bob', 40, 50) sue = Person('Sue', rate=100, hours=60) class Person: ...# Nondecorator approachbob = Wrapper(Person('Bob', 40, 50)) sue = Wrapper(Person('Sue', rate=100, hours=60))
>>>x# 2.XTrace: __repr__ [4, 5, 6, 7] >>>x# 3.X<interfacetracer.Tracer.<locals>.Wrapper object at 0x02946358>
class Tracer:
def __init__(self, aClass): # On @decorator
self.aClass = aClass # Use instance attribute
def __call__(self, *args): # On instance creation
self.wrapped = self.aClass(*args) # ONE (LAST) INSTANCE PER CLASS!
return self
def __getattr__(self, attrname):
print('Trace: ' + attrname)
return getattr(self.wrapped, attrname)
@Tracer # Triggers __init__
class Spam: # Like: Spam = Tracer(Spam)
def display(self):
print('Spam!' * 8)
...
food = Spam() # Triggers __call__
food.display() # Triggers __getattr__@Tracer class Person:# Person = Tracer(Person)def __init__(self, name):# Wrapper bound to Personself.name = name bob = Person('Bob')# bob is really a Wrapperprint(bob.name)# Wrapper embeds a PersonSue = Person('Sue') print(sue.name)# sue overwrites bobprint(bob.name)# OOPS: now bob's name is 'Sue'!
Trace: name Bob Trace: name Sue Trace: name Sue
class Spam:# Nondecorator version...# Any class will dofood = Wrapper(Spam())# Special creation syntax@Tracer class Spam:# Decorator version...# Requires @ syntax at classfood = Spam()# Normal creation syntax
instances = {}
def getInstance(aClass, *args, **kwargs):
if aClass not in instances:
instances[aClass] = aClass(*args, **kwargs)
return instances[aClass]
bob = getInstance(Person, 'Bob', 40, 10) # Versus: bob = Person('Bob', 40, 10)instances = {}
def getInstance(object):
aClass = object.__class__
if aClass not in instances:
instances[aClass] = object
return instances[aClass]
bob = getInstance(Person('Bob', 40, 10)) # Versus: bob = Person('Bob', 40, 10)def func(x, y):# Nondecorator version...# def tracer(func, args): ... func(*args)result = tracer(func, (1, 2))# Special call syntax@tracer def func(x, y):# Decorator version...# Rebinds name: func = tracer(func)result = func(1, 2)# Normal call syntax
As we’ve seen, when wrappers are inserted, a decorated function or class does not retain its original type — it is rebound to a wrapper (proxy) object, which might matter in programs that use object names or test object types. In the singleton example, both the decorator and manager function approaches retain the original class type for instances; in the tracer code, neither approach does, because wrappers are required. Of course, you should avoid type checks in a polymorphic language like Python anyhow, but there are exceptions to most rules.
A wrapping layer added by decoration incurs the additional performance cost of an extra call each time the decorated object is invoked — calls are relatively time-expensive operations, so decoration wrappers can make a program slower. In the tracer code, both approaches require each attribute to be routed through a wrapper layer; the singleton example avoids extra calls by retaining the original class type.
Because decorators augment a function or class, they generally apply to every later call to the decorated object. That ensures uniform deployment, but can also be a negative if you’d rather apply an augmentation more selectively on a call-by-call basis.
Decorators make augmentation explicit and obvious. Their@syntax is easier to recognize than special code in calls that may appear anywhere in a source file — in our singleton and tracer examples, for instance, the decorator lines seem more likely to be noticed than extra code at calls would be. Moreover, decorators allow function and instance creation calls to use normal syntax familiar to all Python programmers.
Decorators avoid repeated augmentation code at each function or class call. Because they appear just once, at the definition of the class or function itself, they obviate redundancy and simplify future code maintenance. For our singleton and tracer cases, we need to use special code at each call to use a manager function approach — extra work is required both initially and for any modifications that must be made in the future.
Decorators make it less likely that a programmer will forget to use required wrapping logic. This derives mostly from the two prior advantages — because decoration is explicit and appears only once, at the decorated objects themselves, decorators promote more consistent and uniform API usage than special code that must be included at each call. In the singleton example, for instance, it would be easy to forget to route all class creation calls through special code, which would subvert the singleton management altogether.
# Registering decorated objects to an APIfrom __future__ import print_function# 2.Xregistry = {} def register(obj):# Both class and func decoratorregistry[obj.__name__] = obj# Add to registryreturn obj# Return obj itself, not a wrapper@register def spam(x): return(x ** 2)# spam = register(spam)@register def ham(x): return(x ** 3) @register class Eggs:# Eggs = register(Eggs)def __init__(self, x): self.data = x ** 4 def __str__(self): return str(self.data) print('Registry:') for name in registry: print(name, '=>', registry[name], type(registry[name])) print('\nManual calls:') print(spam(2))# Invoke objects manuallyprint(ham(2))# Later calls not interceptedX = Eggs(2) print(X) print('\nRegistry calls:') for name in registry: print(name, '=>', registry[name](2))# Invoke from registry
c:\code> py −3 registry-deco.py
Registry:
spam => <function spam at 0x02969158> <class 'function'>
ham => <function ham at 0x02969400> <class 'function'>
Eggs => <class '__main__.Eggs'> <class 'type'>
Manual calls:
4
8
16
Registry calls:
spam => 4
ham => 8
Eggs => 16# Augmenting decorated objects directly>>>def decorate(func):func.marked = True# Assign function attribute for later usereturn func>>>@decoratedef spam(a, b):return a + b>>>spam.markedTrue >>>def annotate(text):# Same, but value is decorator argumentdef decorate(func):func.label = textreturn funcreturn decorate>>>@annotate('spam data')def spam(a, b):# spam = annotate(...)(spam)return a + b>>>spam(1, 2), spam.label(3, 'spam data')
"""
File access1.py (3.X + 2.X)
Privacy for attributes fetched from class instances.
See self-test code at end of file for a usage example.
Decorator same as: Doubler = Private('data', 'size')(Doubler).
Private returns onDecorator, onDecorator returns onInstance,
and each onInstance instance embeds a Doubler instance.
"""
traceMe = False
def trace(*args):
if traceMe: print('[' + ' '.join(map(str, args)) + ']')
def Private(*privates): # privates in enclosing scope
def onDecorator(aClass): # aClass in enclosing scope
class onInstance: # wrapped in instance attribute
def __init__(self, *args, **kargs):
self.wrapped = aClass(*args, **kargs)
def __getattr__(self, attr): # My attrs don't call getattr
trace('get:', attr) # Others assumed in wrapped
if attr in privates:
raise TypeError('private attribute fetch: ' + attr)
else:
return getattr(self.wrapped, attr)
def __setattr__(self, attr, value): # Outside accesses
trace('set:', attr, value) # Others run normally
if attr == 'wrapped': # Allow my attrs
self.__dict__[attr] = value # Avoid looping
elif attr in privates:
raise TypeError('private attribute change: ' + attr)
else:
setattr(self.wrapped, attr, value) # Wrapped obj attrs
return onInstance # Or use __dict__
return onDecorator
if __name__ == '__main__':
traceMe = True
@Private('data', 'size') # Doubler = Private(...)(Doubler)
class Doubler:
def __init__(self, label, start):
self.label = label # Accesses inside the subject class
self.data = start # Not intercepted: run normally
def size(self):
return len(self.data) # Methods run with no checking
def double(self): # Because privacy not inherited
for i in range(self.size()):
self.data[i] = self.data[i] * 2
def display(self):
print('%s => %s' % (self.label, self.data))
X = Doubler('X is', [1, 2, 3])
Y = Doubler('Y is', [-10, −20, −30])
# The following all succeed
print(X.label) # Accesses outside subject class
X.display(); X.double(); X.display() # Intercepted: validated, delegated
print(Y.label)
Y.display(); Y.double()
Y.label = 'Spam'
Y.display()
# The following all fail properly
"""
print(X.size()) # prints "TypeError: private attribute fetch: size"
print(X.data)
X.data = [1, 1, 1]
X.size = lambda S: 0
print(Y.data)
print(Y.size())
""" c:\code> py −3 access1.py
[set: wrapped <__main__.Doubler object at 0x00000000029769B0>]
[set: wrapped <__main__.Doubler object at 0x00000000029769E8>]
[get: label]
X is
[get: display]
X is => [1, 2, 3]
[get: double]
[get: display]
X is => [2, 4, 6]
[get: label]
Y is
[get: display]
Y is => [-10, −20, −30]
[get: double]
[set: label Spam]
[get: display]
Spam => [−20, −40, −60]Private
are used before decoration occurs and are retained as an enclosing
scope reference for use in both onDecorator and onInstance.onDecorator is used at decoration time
and is retained as an enclosing scope reference for use at
instance construction time.onInstance
proxy object, for use when attributes are later accessed from
outside the class.Private declares attributes
of a class’s instances that cannot be fetched
or assigned, except from within the code of the class’s methods.
That is, any name declared Private cannot be accessed from outside
the class, while any name not declared Private can be freely fetched or assigned
from outside the class.Public declares attributes
of a class’s instances that can be fetched or
assigned from both outside the class and within the class’s methods.
That is, any name declared Public
can be freely accessed anywhere, while any name not declared
Public cannot be accessed from
outside the class."""
File access2.py (3.X + 2.X)
Class decorator with Private and Public attribute declarations.
Controls external access to attributes stored on an instance, or
Inherited by it from its classes. Private declares attribute names
that cannot be fetched or assigned outside the decorated class,
and Public declares all the names that can.
Caveat: this works in 3.X for explicitly named attributes only: __X__
operator overloading methods implicitly run for built-in operations
do not trigger either __getattr__ or __getattribute__ in new-style
classes. Add __X__ methods here to intercept and delegate built-ins.
"""
traceMe = False
def trace(*args):
if traceMe: print('[' + ' '.join(map(str, args)) + ']')
def accessControl(failIf):
def onDecorator(aClass):
class onInstance:
def __init__(self, *args, **kargs):
self.__wrapped = aClass(*args, **kargs)
def __getattr__(self, attr):
trace('get:', attr)
if failIf(attr):
raise TypeError('private attribute fetch: ' + attr)
else:
return getattr(self.__wrapped, attr)
def __setattr__(self, attr, value):
trace('set:', attr, value)
if attr == '_onInstance__wrapped':
self.__dict__[attr] = value
elif failIf(attr):
raise TypeError('private attribute change: ' + attr)
else:
setattr(self.__wrapped, attr, value)
return onInstance
return onDecorator
def Private(*attributes):
return accessControl(failIf=(lambda attr: attr in attributes))
def Public(*attributes):
return accessControl(failIf=(lambda attr: attr not in attributes))>>>from access2 import Private, Public>>>@Private('age')# Person = Private('age')(Person)class Person:# Person = onInstance with statedef __init__(self, name, age):self.name = nameself.age = age# Inside accesses run normally>>>X = Person('Bob', 40)>>>X.name# Outside accesses validated'Bob' >>>X.name = 'Sue'>>>X.name'Sue' >>>X.ageTypeError: private attribute fetch: age >>>X.age = 'Tom'TypeError: private attribute change: age >>>@Public('name')class Person:def __init__(self, name, age):self.name = nameself.age = age>>>X = Person('bob', 40)# X is an onInstance>>>X.name# onInstance embeds Person'bob' >>>X.name = 'Sue'>>>X.name'Sue' >>>X.ageTypeError: private attribute fetch: age >>>X.age = 'Tom'TypeError: private attribute change: age
C:\code>c:\python27\python>>>from access2 import Private>>>@Private('age')class Person:def __init__(self):self.age = 42def __str__(self):return 'Person: ' + str(self.age)def __add__(self, yrs):self.age += yrs>>>X = Person()>>>X.age# Name validations fail correctlyTypeError: private attribute fetch: age >>>print(X)# __getattr__ => runs Person.__str__Person: 42 >>>X + 10# __getattr__ => runs Person.__add__>>>print(X)# __getattr__ => runs Person.__str__Person: 52
C:\code>c:\python33\python>>>from access2 import Private>>>@Private('age')class Person:def __init__(self):self.age = 42def __str__(self):return 'Person: ' + str(self.age)def __add__(self, yrs):self.age += yrs>>>X = Person()# Name validations still work>>>X.age# But 3.X fails to delegate built-ins!TypeError: private attribute fetch: age >>>print(X)<access2.accessControl.<locals>.onDecorator.<locals>.onInstance object at...etc> >>>X + 10TypeError: unsupported operand type(s) for +: 'onInstance' and 'int' >>>print(X)<access2.accessControl.<locals>.onDecorator.<locals>.onInstance object at...etc>
>>>X.__add__(10)# Though calls by name work normally>>>X._onInstance__wrapped.age# Break privacy to view result...52
def accessControl(failIf):
def onDecorator(aClass):
class onInstance:
def __init__(self, *args, **kargs):
self.__wrapped = aClass(*args, **kargs)
# Intercept and delegate built-in operations specifically
def __str__(self):
return str(self.__wrapped)
def __add__(self, other):
return self.__wrapped + other # Or getattr(x, '__add__')(y)
def __getitem__(self, index):
return self.__wrapped[index] # If needed
def __call__(self, *args, **kargs):
return self.__wrapped(*args, **kargs) # If needed
# plus any others needed
# Intercept and delegate by-name attribute access generically
def __getattr__(self, attr): ...
def __setattr__(self, attr, value): ...
return onInstance
return onDecorator__getattr__. It requires that operator
overloading names be public per the decorator’s specifications,
but built-in operation calls will work the same as both explicit
name calls and 2.X’s classic classes._wrapped giving access to the embedded
object — which is less than ideal because it precludes wrapped
objects from using the same name and creates a subclass
dependency, but better than using the mangled and class-specific
_onInstance__wrapped, and no
worse than a similarly named method.class BuiltinsMixin:
def __add__(self, other):
return self.__class__.__getattr__(self, '__add__')(other)
def __str__(self):
return self.__class__.__getattr__(self, '__str__')()
def __getitem__(self, index):
return self.__class__.__getattr__(self, '__getitem__')(index)
def __call__(self, *args, **kargs):
return self.__class__.__getattr__(self, '__call__')(*args, **kargs)
# plus any others needed
def accessControl(failIf):
def onDecorator(aClass):
class onInstance(BuiltinsMixin):
...rest unchanged...
def __getattr__(self, attr): ...
def __setattr__(self, attr, value): ...
class BuiltinsMixin:
def __add__(self, other):
return self._wrapped + other # Assume a _wrapped
def __str__(self): # Bypass __getattr__
return str(self._wrapped)
def __getitem__(self, index):
return self._wrapped[index]
def __call__(self, *args, **kargs):
return self._wrapped(*args, **kargs)
# plus any others needed
def accessControl(failIf):
def onDecorator(aClass):
class onInstance(BuiltinsMixin):
...and use self._wrapped instead of self.__wrapped...
def __getattr__(self, attr): ...
def __setattr__(self, attr, value): ...class BuiltinsMixin:
def reroute(self, attr, *args, **kargs):
return self.__class__.__getattr__(self, attr)(*args, **kargs)
def __add__(self, other):
return self.reroute('__add__', other)
def __str__(self):
return self.reroute('__str__')
def __getitem__(self, index):
return self.reroute('__getitem__', index)
def __call__(self, *args, **kargs):
return self.reroute('__call__', *args, **kargs)
# plus any others neededclass BuiltinsMixin:
class ProxyDesc(object): # object for 2.X
def __init__(self, attrname):
self.attrname = attrname
def __get__(self, instance, owner):
return getattr(instance._wrapped, self.attrname) # Assume a _wrapped
builtins = ['add', 'str', 'getitem', 'call'] # Plus any others
for attr in builtins:
exec('__%s__ = ProxyDesc("__%s__")' % (attr, attr)) __add__ = ProxyDesc("__add__")
__str__ = ProxyDesc("__str__")
...etc...__getattr__, the rerouter mix-ins
require either that all __X__ names accessed be listed in Public decorations, or that Private be used instead when operator
overloading is present in clients. In classes that use overloading
heavily, Public may be
impractical.__getattr__ entirely, as coded here both
the inline scheme and self._wrapped mix-ins do not have these
constraints, but they preclude built-in operations from being made
private, and cause built-in operation dispatch to work
asymmetrically from both explicit __X__ calls by-name and 2.X’s default
classic classes.__X__ names are routed through __getattr__ automatically.Public decorators may
need to list names from both lines).# Method insertion: rest of access2.py code as beforedef accessControl(failIf): def onDecorator(aClass): def getattributes(self, attr): trace('get:', attr) if failIf(attr): raise TypeError('private attribute fetch: ' + attr) else: return object.__getattribute__(self, attr) def setattributes(self, attr, value): trace('set:', attr) if failIf(attr): raise TypeError('private attribute change: ' + attr) else: return object.__setattr__(self, attr, value) aClass.__getattribute__ = getattributes aClass.__setattr__ = setattributes# Insert accessorsreturn aClass# Return original classreturn onDecorator
class Person:
...
def giveRaise(self, percent):
self.pay = int(self.pay * (1 + percent))class Person:
def giveRaise(self, percent): # Validate with inline code
if percent < 0.0 or percent > 1.0:
raise TypeError, 'percent invalid'
self.pay = int(self.pay * (1 + percent))
class Person: # Validate with asserts
def giveRaise(self, percent):
assert percent >= 0.0 and percent <= 1.0, 'percent invalid'
self.pay = int(self.pay * (1 + percent))class Person:
@rangetest(percent=(0.0, 1.0)) # Use decorator to validate
def giveRaise(self, percent):
self.pay = int(self.pay * (1 + percent))def rangetest(*argchecks):# Validate positional arg rangesdef onDecorator(func): if not __debug__:# True if "python -O main.py args..."return func# No-op: call original directlyelse:# Else wrapper while debuggingdef onCall(*args): for (ix, low, high) in argchecks: if args[ix] < low or args[ix] > high: errmsg = 'Argument %s not in %s..%s' % (ix, low, high) raise TypeError(errmsg) return func(*args) return onCall return onDecorator
# File rangetest1_test.pyfrom __future__ import print_function # 2.X from rangetest1 import rangetest print(__debug__)# False if "python -O main.py"@rangetest((1, 0, 120))# persinfo = rangetest(...)(persinfo)def persinfo(name, age):# age must be in 0..120print('%s is %s years old' % (name, age)) @rangetest([0, 1, 12], [1, 1, 31], [2, 0, 2009]) def birthday(M, D, Y): print('birthday = {0}/{1}/{2}'.format(M, D, Y)) class Person: def __init__(self, name, job, pay): self.job = job self.pay = pay @rangetest([1, 0.0, 1.0])# giveRaise = rangetest(...)(giveRaise)def giveRaise(self, percent):# Arg 0 is the self instance hereself.pay = int(self.pay * (1 + percent))# Comment lines raise TypeError unless "python -O" used on shell command linepersinfo('Bob Smith', 45)# Really runs onCall(...) with state#persinfo('Bob Smith', 200)# Or person if -O cmd line argumentbirthday(5, 31, 1963) #birthday(5, 32, 1963) sue = Person('Sue Jones', 'dev', 100000) sue.giveRaise(.10)# Really runs onCall(self, .10)print(sue.pay)# Or giveRaise(self, .10) if -O#sue.giveRaise(1.10) #print(sue.pay)
C:\code> python rangetest1_test.py
True
Bob Smith is 45 years old
birthday = 5/31/1963
110000C:\code> python rangetest1_test.py
True
Bob Smith is 45 years old
birthday = 5/31/1963
110000
TypeError: Argument 1 not in 0.0..1.0C:\code> python -O rangetest1_test.py
False
Bob Smith is 45 years old
birthday = 5/31/1963
110000
231000""" File rangetest.py: function decorator that performs range-test validation for arguments passed to any function or method. Arguments are specified by keyword to the decorator. In the actual call, arguments may be passed by position or keyword, and defaults may be omitted. See rangetest_test.py for example use cases. """ trace = True def rangetest(**argchecks):# Validate ranges for both+defaultsdef onDecorator(func):# onCall remembers func and argchecksif not __debug__:# True if "python -O main.py args..."return func# Wrap if debugging; else use originalelse: code = func.__code__ allargs = code.co_varnames[:code.co_argcount] funcname = func.__name__ def onCall(*pargs, **kargs):# All pargs match first N expected args by position# The rest must be in kargs or be omitted defaultsexpected = list(allargs) positionals = expected[:len(pargs)] for (argname, (low, high)) in argchecks.items():# For all args to be checkedif argname in kargs:# Was passed by nameif kargs[argname] < low or kargs[argname] > high: errmsg = '{0} argument "{1}" not in {2}..{3}' errmsg = errmsg.format(funcname, argname, low, high) raise TypeError(errmsg) elif argname in positionals:# Was passed by positionposition = positionals.index(argname) if pargs[position] < low or pargs[position] > high: errmsg = '{0} argument "{1}" not in {2}..{3}' errmsg = errmsg.format(funcname, argname, low, high) raise TypeError(errmsg) else:# Assume not passed: defaultif trace: print('Argument "{0}" defaulted'.format(argname)) return func(*pargs, **kargs)# OK: run original callreturn onCall return onDecorator
""" File rangetest_test.py (3.X + 2.X) Comment lines raise TypeError unless "python -O" used on shell command line """ from __future__ import print_function # 2.X from rangetest import rangetest# Test functions, positional and keyword@rangetest(age=(0, 120))# persinfo = rangetest(...)(persinfo)def persinfo(name, age): print('%s is %s years old' % (name, age)) @rangetest(M=(1, 12), D=(1, 31), Y=(0, 2013)) def birthday(M, D, Y): print('birthday = {0}/{1}/{2}'.format(M, D, Y)) persinfo('Bob', 40) persinfo(age=40, name='Bob') birthday(5, D=1, Y=1963) #persinfo('Bob', 150) #persinfo(age=150, name='Bob') #birthday(5, D=40, Y=1963)# Test methods, positional and keywordclass Person: def __init__(self, name, job, pay): self.job = job self.pay = pay# giveRaise = rangetest(...)(giveRaise)@rangetest(percent=(0.0, 1.0))# percent passed by name or positiondef giveRaise(self, percent): self.pay = int(self.pay * (1 + percent)) bob = Person('Bob Smith', 'dev', 100000) sue = Person('Sue Jones', 'dev', 100000) bob.giveRaise(.10) sue.giveRaise(percent=.20) print(bob.pay, sue.pay) #bob.giveRaise(1.10) #bob.giveRaise(percent=1.20)# Test omitted defaults: skipped@rangetest(a=(1, 10), b=(1, 10), c=(1, 10), d=(1, 10)) def omitargs(a, b=7, c=8, d=9): print(a, b, c, d) omitargs(1, 2, 3, 4) omitargs(1, 2, 3) omitargs(1, 2, 3, d=4) omitargs(1, d=4) omitargs(d=4, a=1) omitargs(1, b=2, d=4) omitargs(d=8, c=7, a=1) #omitargs(1, 2, 3, 11)# Bad d#omitargs(1, 2, 11)# Bad c#omitargs(1, 2, 3, d=11)# Bad d#omitargs(11, d=4)# Bad a#omitargs(d=4, a=11)# Bad a#omitargs(1, b=11, d=4)# Bad b#omitargs(d=8, c=7, a=11)# Bad a
C:\code> python rangetest_test.py
Bob is 40 years old
Bob is 40 years old
birthday = 5/1/1963
110000 120000
1 2 3 4
Argument "d" defaulted
1 2 3 9
1 2 3 4
Argument "c" defaulted
Argument "b" defaulted
1 7 8 4
Argument "c" defaulted
Argument "b" defaulted
1 7 8 4
Argument "c" defaulted
1 2 8 4
Argument "b" defaulted
1 7 7 8TypeError: giveRaise argument "percent" not in 0.0..1.0
# In Python 3.X (and 2.6+ for compatibility)>>>def func(a, b, c, e=True, f=None):# Args: three required, two defaultsx = 1# Plus two more local variablesy = 2>>>code = func.__code__# Code object of function object>>>code.co_nlocals7 >>>code.co_varnames# All local variable names('a', 'b', 'c', 'e', 'f', 'x', 'y') >>>code.co_varnames[:code.co_argcount]# <== First N locals are expected args('a', 'b', 'c', 'e', 'f')
>>>def catcher(*pargs, **kargs): print('%s, %s' % (pargs, kargs))>>>catcher(1, 2, 3, 4, 5)(1, 2, 3, 4, 5), {} >>>catcher(1, 2, c=3, d=4, e=5)# Arguments at calls(1, 2), {'d': 4, 'e': 5, 'c': 3}
>>>import sys# For backward compatibility>>>tuple(sys.version_info)# [0] is major release number(3, 3, 0, 'final', 0) >>>code = func.__code__ if sys.version_info[0] == 3 else func.func_code
def, all
nondefault arguments appear before all default arguments.N be the number of passed positional arguments, obtained from the
length of the *pargs
tuple.*pargs must match the first
N expected arguments obtained from the
function’s code object. This is true per Python’s call ordering
rules, outlined earlier, since all positionals precede all
keywords in a call.*pargs passed positionals tuple.**kargs, it was passed by
name — indexing **kargs gives
its passed value.*pargs.omitargs() omitargs(d=8, c=7, b=6)
**kargs and
can be tested normally if mentioned to the decorator.**kargs or
the sliced expected positionals list, and it will thus not be
checked — it is treated as though it were defaulted, even though it
is really an optional extra argument.**kargs or
the sliced expected arguments list, so it will simply be skipped.
Because such arguments are not listed in the function’s
definition, there’s no way to map a name given to the decorator
back to an expected relative position.>>>def func(*kargs, **pargs): pass>>>code = func.__code__>>>code.co_nlocals, code.co_varnames(2, ('kargs', 'pargs')) >>>code.co_argcount, code.co_varnames[:code.co_argcount](0, ()) >>>def func(a, b, *kargs, **pargs): pass>>>code = func.__code__>>>code.co_argcount, code.co_varnames[:code.co_argcount](2, ('a', 'b'))
@rangetest(a=(1, 5), c=(0.0, 1.0))
def func(a, b, c): # func = rangetest(...)(func)
print(a + b + c)@rangetest
def func(a:(1, 5), b, c:(0.0, 1.0)):
print(a + b + c)# Using decorator arguments (3.X + 2.X)def rangetest(**argchecks): def onDecorator(func): def onCall(*pargs, **kargs): print(argchecks) for check in argchecks: pass# Add validation code herereturn func(*pargs, **kargs) return onCall return onDecorator @rangetest(a=(1, 5), c=(0.0, 1.0)) def func(a, b, c):# func = rangetest(...)(func) print(a + b + c) func(1, 2, c=3)# Runs onCall, argchecks in scope# Using function annotations (3.X only)def rangetest(func): def onCall(*pargs, **kargs): argchecks = func.__annotations__ print(argchecks) for check in argchecks: pass# Add validation code herereturn func(*pargs, **kargs) return onCall @rangetest def func(a:(1, 5), b, c:(0.0, 1.0)):# func = rangetest(func)print(a + b + c) func(1, 2, c=3)# Runs onCall, annotations on func
C:\code> py −3 decoargs-vs-annotation.py
{'a': (1, 5), 'c': (0.0, 1.0)}
6
{'a': (1, 5), 'c': (0.0, 1.0)}
6def typetest(**argchecks):
def onDecorator(func):
...
def onCall(*pargs, **kargs):
positionals = list(allargs)[:len(pargs)]
for (argname, type) in argchecks.items():
if argname in kargs:
if not isinstance(kargs[argname], type):
...
raise TypeError(errmsg)
elif argname in positionals:
position = positionals.index(argname)
if not isinstance(pargs[position], type):
...
raise TypeError(errmsg)
else:
# Assume not passed: default
return func(*pargs, **kargs)
return onCall
return onDecorator
@typetest(a=int, c=float)
def func(a, b, c, d): # func = typetest(...)(func)
...
func(1, 2, 3.0, 4) # OK
func('spam', 2, 99, 4) # Triggers exception correctly@typetest def func(a: int, b, c: float, d):# func = typetest(func)...# Gasp!...
__call__ operator overloading
method to catch calls. This structure does not work for a class’s
methods because the decorator instance is passed
to self, not the subject class
instance.Public/Private class decorators we wrote in module
access2.py in this chapter’s
first case study example will add performance
costs to every attribute fetch in a decorated class.
Although we could simply delete the @ decoration line to gain speed, we could
also augment the decorator itself to check the __debug__ switch and perform no wrapping at
all when the –O Python flag is
passed on the command line — just as we did for the argument range-test
decorators. That way, we can speed our program without changing its
source, via command-line arguments (python –O
main.py...). While we’re at it, we could also use one of the
mix-in superclass techniques we studied to catch a few
built-in operations in Python 3.X too. Code and
test these two extensions.self argument is not the decorator’s
instance, and assigning the total time to the decorator function
itself so it can be fetched later through the original rebound name
(see the section “State Information Retention Options” of this chapter
for details — functions support arbitrary attribute attachment, and the
function name is an enclosing scope reference in this context). If you
wish to expand this further, it might be useful to also record the
best (minimum) call time in addition to the total
time, as we did in Chapter 21’s
timer examples.""" File timerdeco.py (3.X + 2.X) Call timer decorator for both functions and methods. """ import time def timer(label='', trace=True):# On decorator args: retain argsdef onDecorator(func):# On @: retain decorated funcdef onCall(*args, **kargs):# On calls: call originalstart = time.clock()# State is scopes + func attrresult = func(*args, **kargs) elapsed = time.clock() - start onCall.alltime += elapsed if trace: format = '%s%s: %.5f, %.5f' values = (label, func.__name__, elapsed, onCall.alltime) print(format % values) return result onCall.alltime = 0 return onCall return onDecorator
"""
File timerdeco-test.py
"""
from __future__ import print_function # 2.X
from timerdeco import timer
import sys
force = list if sys.version_info[0] == 3 else (lambda X: X)
print('---------------------------------------------------')
# Test on functions
@timer(trace=True, label='[CCC]==>')
def listcomp(N): # Like listcomp = timer(...)(listcomp)
return [x * 2 for x in range(N)] # listcomp(...) triggers onCall
@timer('[MMM]==>')
def mapcall(N):
return force(map((lambda x: x * 2), range(N))) # list() for 3.X views
for func in (listcomp, mapcall):
result = func(5) # Time for this call, all calls, return value
func(5000000)
print(result)
print('allTime = %s\n' % func.alltime) # Total time for all calls
print('---------------------------------------------------')
# Test on methods
class Person:
def __init__(self, name, pay):
self.name = name
self.pay = pay
@timer()
def giveRaise(self, percent): # giveRaise = timer()(giveRaise)
self.pay *= (1.0 + percent) # tracer remembers giveRaise
@timer(label='**')
def lastName(self): # lastName = timer(...)(lastName)
return self.name.split()[-1] # alltime per class, not instance
bob = Person('Bob Smith', 50000)
sue = Person('Sue Jones', 100000)
bob.giveRaise(.10)
sue.giveRaise(.20) # runs onCall(sue, .10)
print(int(bob.pay), int(sue.pay))
print(bob.lastName(), sue.lastName()) # runs onCall(bob), remembers lastName
print('%.5f %.5f' % (Person.giveRaise.alltime, Person.lastName.alltime))c:\code> py −3 timerdeco-test.py
---------------------------------------------------
[CCC]==>listcomp: 0.00001, 0.00001
[CCC]==>listcomp: 0.57930, 0.57930
[0, 2, 4, 6, 8]
allTime = 0.5793010457092784
[MMM]==>mapcall: 0.00002, 0.00002
[MMM]==>mapcall: 1.08609, 1.08611
[0, 2, 4, 6, 8]
allTime = 1.0861149923442373
---------------------------------------------------
giveRaise: 0.00001, 0.00001
giveRaise: 0.00000, 0.00001
55000 120000
**lastName: 0.00001, 0.00001
**lastName: 0.00000, 0.00001
Smith Jones
0.00001 0.00001–O), so attribute accesses don’t incur a
speed hit. Mostly, it just adds the debug mode test statements and
indents the class further to the right:"""
File access.py (3.X + 2.X)
Class decorator with Private and Public attribute declarations.
Controls external access to attributes stored on an instance, or
inherited by it from its classes in any fashion.
Private declares attribute names that cannot be fetched or assigned
outside the decorated class, and Public declares all the names that can.
Caveats: in 3.X catches built-ins coded in BuiltinMixins only (expand me);
as coded, Public may be less useful than Private for operator overloading.
"""
from access_builtins import BuiltinsMixin # A partial set!
traceMe = False
def trace(*args):
if traceMe: print('[' + ' '.join(map(str, args)) + ']')
def accessControl(failIf):
def onDecorator(aClass):
if not __debug__:
return aClass
else:
class onInstance(BuiltinsMixin):
def __init__(self, *args, **kargs):
self.__wrapped = aClass(*args, **kargs)
def __getattr__(self, attr):
trace('get:', attr)
if failIf(attr):
raise TypeError('private attribute fetch: ' + attr)
else:
return getattr(self.__wrapped, attr)
def __setattr__(self, attr, value):
trace('set:', attr, value)
if attr == '_onInstance__wrapped':
self.__dict__[attr] = value
elif failIf(attr):
raise TypeError('private attribute change: ' + attr)
else:
setattr(self.__wrapped, attr, value)
return onInstance
return onDecorator
def Private(*attributes):
return accessControl(failIf=(lambda attr: attr in attributes))
def Public(*attributes):
return accessControl(failIf=(lambda attr: attr not in attributes))__getattr__ already, but in 3.X is a
new-style class that does not. The mix-in used here requires listing
such methods in Public decorators;
see earlier for alternatives that do not (but that also do not allow
built-ins to be made private), and expand this class as needed:"""
File access_builtins.py (from access2_builtins2b.py)
Route some built-in operations back to proxy class __getattr__, so they
work the same in 3.X as direct by-name calls and 2.X's default classic classes.
Expand me as needed to include other __X__ names used by proxied objects.
"""
class BuiltinsMixin:
def reroute(self, attr, *args, **kargs):
return self.__class__.__getattr__(self, attr)(*args, **kargs)
def __add__(self, other):
return self.reroute('__add__', other)
def __str__(self):
return self.reroute('__str__')
def __getitem__(self, index):
return self.reroute('__getitem__', index)
def __call__(self, *args, **kargs):
return self.reroute('__call__', *args, **kargs)
# Plus any others used by wrapped objects in 3.X only__name__ test and indenting:"""
File: access-test.py
Test code: separate file to allow decorator reuse.
"""
import sys
from access import Private, Public
print('---------------------------------------------------------')
# Test 1: names are public if not private
@Private('age') # Person = Private('age')(Person)
class Person: # Person = onInstance with state
def __init__(self, name, age):
self.name = name
self.age = age # Inside accesses run normally
def __add__(self, N):
self.age += N # Built-ins caught by mix-in in 3.X
def __str__(self):
return '%s: %s' % (self.name, self.age)
X = Person('Bob', 40)
print(X.name) # Outside accesses validated
X.name = 'Sue'
print(X.name)
X + 10
print(X)
try: t = X.age # FAILS unless "python -O"
except: print(sys.exc_info()[1])
try: X.age = 999 # ditto
except: print(sys.exc_info()[1])
print('---------------------------------------------------------')
# Test 2: names are private if not public
# Operators must be non-Private or Public in BuiltinMixin used
@Public('name', '__add__', '__str__', '__coerce__')
class Person:
def __init__(self, name, age):
self.name = name
self.age = age
def __add__(self, N):
self.age += N # Built-ins caught by mix-in in 3.X
def __str__(self):
return '%s: %s' % (self.name, self.age)
X = Person('bob', 40) # X is an onInstance
print(X.name) # onInstance embeds Person
X.name = 'sue'
print(X.name)
X + 10
print(X)
try: t = X.age # FAILS unless "python -O"
except: print(sys.exc_info()[1])
try: X.age = 999 # ditto
except: print(sys.exc_info()[1])Private and
then with Public:c:\code>py −3 access-test.py--------------------------------------------------------- Bob Sue Sue: 50 private attribute fetch: age private attribute change: age --------------------------------------------------------- bob sue sue: 50 private attribute fetch: age private attribute change: age c:\code>py −3 -O access-test.py# Suppresses the four access error messages
valuetest can be used
to combine differing types of tests in a single decoration (though the
amount of code needed in this mode may negate much of its benefits
over a simple assert!)."""
File argtest.py: (3.X + 2.X) function decorator that performs
arbitrary passed-in validations for arguments passed to any
function method. Range and type tests are two example uses;
valuetest handles more arbitrary tests on an argument's value.
Arguments are specified by keyword to the decorator. In the actual
call, arguments may be passed by position or keyword, and defaults
may be omitted. See self-test code below for example use cases.
Caveats: doesn't fully support nesting because call proxy args
differ; doesn't validate extra args passed to a decoratee's *args;
and may be no easier than an assert except for canned use cases.
"""
trace = False
def rangetest(**argchecks):
return argtest(argchecks, lambda arg, vals: arg < vals[0] or arg > vals[1])
def typetest(**argchecks):
return argtest(argchecks, lambda arg, type: not isinstance(arg, type))
def valuetest(**argchecks):
return argtest(argchecks, lambda arg, tester: not tester(arg))
def argtest(argchecks, failif): # Validate args per failif + criteria
def onDecorator(func): # onCall retains func, argchecks, failif
if not __debug__: # No-op if "python -O main.py args..."
return func
else:
code = func.__code__
expected = list(code.co_varnames[:code.co_argcount])
def onError(argname, criteria):
errfmt = '%s argument "%s" not %s'
raise TypeError(errfmt % (func.__name__, argname, criteria))
def onCall(*pargs, **kargs):
positionals = expected[:len(pargs)]
for (argname, criteria) in argchecks.items(): # For all to test
if argname in kargs: # Passed by name
if failif(kargs[argname], criteria):
onError(argname, criteria)
elif argname in positionals: # Passed by posit
position = positionals.index(argname)
if failif(pargs[position], criteria):
onError(argname, criteria)
else: # Not passed-dflt
if trace:
print('Argument "%s" defaulted' % argname)
return func(*pargs, **kargs) # OK: run original call
return onCall
return onDecorator
if __name__ == '__main__':
import sys
def fails(test):
try: result = test()
except: print('[%s]' % sys.exc_info()[1])
else: print('?%s?' % result)
print('--------------------------------------------------------------------')
# Canned use cases: ranges, types
@rangetest(m=(1, 12), d=(1, 31), y=(1900, 2013))
def date(m, d, y):
print('date = %s/%s/%s' % (m, d, y))
date(1, 2, 1960)
fails(lambda: date(1, 2, 3))
@typetest(a=int, c=float)
def sum(a, b, c, d):
print(a + b + c + d)
sum(1, 2, 3.0, 4)
sum(1, d=4, b=2, c=3.0)
fails(lambda: sum('spam', 2, 99, 4))
fails(lambda: sum(1, d=4, b=2, c=99))
print('--------------------------------------------------------------------')
# Arbitrary/mixed tests
@valuetest(word1=str.islower, word2=(lambda x: x[0].isupper()))
def msg(word1='mighty', word2='Larch', label='The'):
print('%s %s %s' % (label, word1, word2))
msg() # word1 and word2 defaulted
msg('majestic', 'Moose')
fails(lambda: msg('Giant', 'Redwood'))
fails(lambda: msg('great', word2='elm'))
print('--------------------------------------------------------------------')
# Manual type and range tests
@valuetest(A=lambda x: isinstance(x, int), B=lambda x: x > 0 and x < 10)
def manual(A, B):
print(A + B)
manual(100, 2)
fails(lambda: manual(1.99, 2))
fails(lambda: manual(100, 20))
print('--------------------------------------------------------------------')
# Nesting: runs both, by nesting proxies on original.
# Open issue: outer levels do not validate positionals due
# to call proxy function's differing argument signature;
# when trace=True, in all but the last of these "X" is
# classified as defaulted due to the proxy's signature.
@rangetest(X=(1, 10))
@typetest(Z=str) # Only innermost validates positional args
def nester(X, Y, Z):
return('%s-%s-%s' % (X, Y, Z))
print(nester(1, 2, 'spam')) # Original function runs properly
fails(lambda: nester(1, 2, 3)) # Nested typetest is run: positional
fails(lambda: nester(1, 2, Z=3)) # Nested typetest is run: keyword
fails(lambda: nester(0, 2, 'spam')) # <==Outer rangetest not run: posit.
fails(lambda: nester(X=0, Y=2, Z='spam')) # Outer rangetest is run: keywordc:\code> py −3 argtest.py
--------------------------------------------------------------------
date = 1/2/1960
[date argument "y" not (1900, 2013)]
10.0
10.0
[sum argument "a" not <class 'int'>]
[sum argument "c" not <class 'float'>]
--------------------------------------------------------------------
The mighty Larch
The majestic Moose
[msg argument "word1" not <method 'islower' of 'str' objects>]
[msg argument "word2" not <function <lambda> at 0x0000000002A096A8>]
--------------------------------------------------------------------
102
[manual argument "A" not <function <lambda> at 0x0000000002A09950>]
[manual argument "B" not <function <lambda> at 0x0000000002A09B70>]
--------------------------------------------------------------------
1-2-spam
[nester argument "Z" not <class 'str'>]
[nester argument "Z" not <class 'str'>]
?0-2-spam?
[onCall argument "X" not (1, 10)]# File argtest_testmeth.pyfrom argtest import rangetest, typetest class C: @rangetest(a=(1, 10)) def meth1(self, a): return a * 1000 @typetest(a=int) def meth2(self, a): return a * 1000 >>>from argtest_testmeth import C>>>X = C()>>>X.meth1(5)5000 >>>X.meth1(20)TypeError: meth1 argument "a" not (1, 10) >>>X.meth2(20)20000 >>>X.meth2(20.9)TypeError: meth2 argument "a" not <class 'int'>
[Metaclasses] are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).
Special attributes like__class__and__dict__allow us to inspect internal implementation aspects of Python objects, in order to process them generically — to list all attributes of an object, display a class’s name, and so on. As we’ve also seen, tools such asdirandgetattrcan serve similar roles when “virtual” attributes such as slots must be supported.
Specially named methods such as__str__and__add__coded in classes intercept and provide behavior for built-in operations applied to class instances, such as printing, expression operators, and so on. They are run automatically in response to built-in operations and allow classes to conform to expected interfaces.
A special category of operator overloading methods provides a way to intercept attribute accesses on instances generically:__getattr__,__setattr__,__delattr__, and__getattribute__allow wrapper (a.k.a. proxy) classes to insert automatically run code that may validate attribute requests and delegate them to embedded objects. They allow any number of attributes of an object to be computed when accessed — either selected attributes, or all of them.
Thepropertybuilt-in allows us to associate code with a specific class attribute that is automatically run when the attribute is fetched, assigned, or deleted. Though not as generic as the prior paragraph’s tools, properties allow for automatic code invocation on access to specific attributes.
Really,propertyis a succinct way to define an attribute descriptor that runs functions on access automatically. Descriptors allow us to code in a separate class__get__,__set__, and__delete__handler methods that are run automatically when an attribute assigned to an instance of that class is accessed. They provide a general way to insert arbitrary code that is run implicitly when a specific attribute is accessed as part of the normal attribute lookup procedure.
As we saw in Chapter 39, the special@callablesyntax for decorators allows us to add logic to be automatically run when a function is called or a class instance is created. This wrapper logic can trace or time calls, validate arguments, manage all instances of a class, augment instances with extra behavior such as attribute fetch validation, and more. Decorator syntax inserts name-rebinding logic to be run at the end of function and class definition statements — decorated function and class names may be rebound to either augmented original objects, or to object proxies that intercept later calls.
The last topic of magic introduced in Chapter 32, which we take up here.
class Extras:
def extra(self, args): # Normal inheritance: too static
...
class Client1(Extras): ... # Clients inherit extra methods
class Client2(Extras): ...
class Client3(Extras): ...
X = Client1() # Make an instance
X.extra() # Run the extra methodsdef extra(self, arg): ...
class Client1: ... # Client augments: too distributed
if required():
Client1.extra = extra
class Client2: ...
if required():
Client2.extra = extra
class Client3: ...
if required():
Client3.extra = extra
X = Client1()
X.extra()def extra(self, arg): ...
def extras(Class): # Manager function: too manual
if required():
Class.extra = extra
class Client1: ...
extras(Client1)
class Client2: ...
extras(Client2)
class Client3: ...
extras(Client3)
X = Client1()
X.extra()def extra(self, arg): ...
class Extras(type):
def __init__(Class, classname, superclasses, attributedict):
if required():
Class.extra = extra
class Client1(metaclass=Extras): ... # Metaclass declaration only (3.X form)
class Client2(metaclass=Extras): ... # Client class is instance of meta
class Client3(metaclass=Extras): ...
X = Client1() # X is instance of Client1
X.extra()def extra(self, arg): ...
def extras(Class):
if required():
Class.extra = extra
return Class
class Client1: ...
Client1 = extras(Client1)
class Client2: ...
Client2 = extras(Client2)
class Client3: ...
Client3 = extras(Client3)
X = Client1()
X.extra()def extra(self, arg): ...
def extras(Class):
if required():
Class.extra = extra
return Class
@extras
class Client1: ... # Client1 = extras(Client1)
@extras
class Client2: ... # Rebinds class independent of instances
@extras
class Client3: ...
X = Client1() # Makes instance of augmented class
X.extra() # X is instance of original Client1type, which is itself a class.object, which is a subclass
of type; classic classes are
instances of type and are not
created from a class.C:\code>py −3# In 3.X:>>>type([]), type(type([]))# List instance is created from list class(<class 'list'>, <class 'type'>)# List class is created from type class>>>type(list), type(type)# Same, but with type names(<class 'type'>, <class 'type'>)# Type of type is type: top of hierarchy
C:\code>py −2>>>type([]), type(type([]))# In 2.X, type is a bit different(<type 'list'>, <type 'type'>) >>>type(list), type(type)(<type 'type'>, <type 'type'>)
type.C:\code>py −3>>>class C: pass# 3.X class object (new-style)>>>X = C()# Class instance object>>>type(X)# Instance is instance of class<class '__main__.C'> >>>X.__class__# Instance's class<class '__main__.C'> >>>type(C)# Class is instance of type<class 'type'> >>>C.__class__# Class's class is type<class 'type'>
C:\code>py −2>>>class C(object): pass# In 2.X new-style classes,>>>X = C()# classes have a class too>>>type(X)<class '__main__.C'> >>>X.__class__<class '__main__.C'> >>>type(C)<type 'type'> >>>C.__class__<type 'type'>
C:\code>py −2>>>class C: pass# In 2.X classic classes,>>>X = C()# classes have no class themselves>>>type(X)<type 'instance'> >>>X.__class__<class __main__.C at 0x005F85A0> >>>type(C)<type 'classobj'> >>>C.__class__AttributeError: class C has no attribute '__class__'
type is a class that
generates user-defined classes.type class.type class, or a subclass thereof.class= type(classname,superclasses,attributedict)
type.__new__(typeclass,classname,superclasses,attributedict) type.__init__(class,classname,superclasses,attributedict)
class Eggs: ...# Inherited names hereclass Spam(Eggs):# Inherits from Eggsdata = 1# Class data attributedef meth(self, arg):# Class method attributereturn self.data + arg
Spam = type('Spam', (Eggs,), {'data': 1, 'meth': meth, '__module__': '__main__'})>>>x = type('Spam', (), {'data': 1, 'meth': (lambda x, y: x.data + y)})>>>i = x()>>>x, i(<class '__main__.Spam'>, <__main__.Spam object at 0x029E7780>) >>>i.data, i.meth(2)(1, 3)
>>>x.__bases__(<class 'object'>,) >>>[(a, v) for (a, v) in x.__dict__.items() if not a.startswith('__')][('data', 1), ('meth', <function <lambda> at 0x0297A158>)]
class Spam(metaclass=Meta): # 3.X version (only)class Spam(Eggs, metaclass=Meta): # Normal supers OK: must list firstclass Spam(object):# 2.X version (only), object optional?__metaclass__ = Meta class Spam(Eggs, object):# Normal supers OK: object suggested__metaclass__ = Meta
class = Meta(classname,superclasses,attributedict)
Meta.__new__(Meta,classname,superclasses,attributedict) Meta.__init__(class,classname,superclasses,attributedict)
class Spam(Eggs, metaclass=Meta):# Inherits from Eggs, instance of Metadata = 1# Class data attributedef meth(self, arg):# Class method attributereturn self.data + arg
Spam = Meta('Spam', (Eggs,), {'data': 1, 'meth': meth, '__module__': '__main__'})class Meta(type):
def __new__(meta, classname, supers, classdict):
# Run by inherited type.__call__
return type.__new__(meta, classname, supers, classdict)class MetaOne(type):
def __new__(meta, classname, supers, classdict):
print('In MetaOne.new:', meta, classname, supers, classdict, sep='\n...')
return type.__new__(meta, classname, supers, classdict)
class Eggs:
pass
print('making class')
class Spam(Eggs, metaclass=MetaOne): # Inherits from Eggs, instance of MetaOne
data = 1 # Class data attribute
def meth(self, arg): # Class method attribute
return self.data + arg
print('making instance')
X = Spam()
print('data:', X.data, X.meth(2))c:\code> py −3 metaclass1.py
making class
In MetaOne.new:
...<class '__main__.MetaOne'>
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x02A191E0>, '__module__': '__main__'}
making instance
data: 1 3from __future__ import print_function# To run the same in 2.X (only)class Eggs(object):# One of the "object" optionalclass Spam(Eggs, object): __metaclass__ = MetaOne
class MetaTwo(type):
def __new__(meta, classname, supers, classdict):
print('In MetaTwo.new: ', classname, supers, classdict, sep='\n...')
return type.__new__(meta, classname, supers, classdict)
def __init__(Class, classname, supers, classdict):
print('In MetaTwo.init:', classname, supers, classdict, sep='\n...')
print('...init class object:', list(Class.__dict__.keys()))
class Eggs:
pass
print('making class')
class Spam(Eggs, metaclass=MetaTwo): # Inherits from Eggs, instance of MetaTwo
data = 1 # Class data attribute
def meth(self, arg): # Class method attribute
return self.data + arg
print('making instance')
X = Spam()
print('data:', X.data, X.meth(2))c:\code> py −3 metaclass2.py
making class
In MetaTwo.new:
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x02967268>, '__module__': '__main__'}
In MetaTwo.init:
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x02967268>, '__module__': '__main__'}
...init class object: ['__qualname__', 'data', '__module__', 'meth', '__doc__']
making instance
data: 1 3# A simple function can serve as a metaclass toodef MetaFunc(classname, supers, classdict): print('In MetaFunc: ', classname, supers, classdict, sep='\n...') return type(classname, supers, classdict) class Eggs: pass print('making class') class Spam(Eggs, metaclass=MetaFunc):# Run simple function at enddata = 1# Function returns classdef meth(self, arg): return self.data + arg print('making instance') X = Spam() print('data:', X.data, X.meth(2))
c:\code> py −3 metaclass3.py
making class
In MetaFunc:
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x029471E0>, '__module__': '__main__'}
making instance
data: 1 3# A normal class instance can serve as a metaclass tooclass MetaObj: def __call__(self, classname, supers, classdict): print('In MetaObj.call: ', classname, supers, classdict, sep='\n...') Class = self.__New__(classname, supers, classdict) self.__Init__(Class, classname, supers, classdict) return Class def __New__(self, classname, supers, classdict): print('In MetaObj.new: ', classname, supers, classdict, sep='\n...') return type(classname, supers, classdict) def __Init__(self, Class, classname, supers, classdict): print('In MetaObj.init:', classname, supers, classdict, sep='\n...') print('...init class object:', list(Class.__dict__.keys())) class Eggs: pass print('making class') class Spam(Eggs, metaclass=MetaObj()):# MetaObj is normal class instancedata = 1# Called at end of statementdef meth(self, arg): return self.data + arg print('making instance') X = Spam() print('data:', X.data, X.meth(2))
c:\code> py −3 metaclass4.py
making class
In MetaObj.call:
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x029492F0>, '__module__': '__main__'}
In MetaObj.new:
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x029492F0>, '__module__': '__main__'}
In MetaObj.init:
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x029492F0>, '__module__': '__main__'}
...init class object: ['__module__', '__doc__', 'data', '__qualname__', 'meth']
making instance
data: 1 3# Instances inherit from classes and their supers normallyclass SuperMetaObj: def __call__(self, classname, supers, classdict): print('In SuperMetaObj.call: ', classname, supers, classdict, sep='\n...') Class = self.__New__(classname, supers, classdict) self.__Init__(Class, classname, supers, classdict) return Class class SubMetaObj(SuperMetaObj): def __New__(self, classname, supers, classdict): print('In SubMetaObj.new: ', classname, supers, classdict, sep='\n...') return type(classname, supers, classdict) def __Init__(self, Class, classname, supers, classdict): print('In SubMetaObj.init:', classname, supers, classdict, sep='\n...') print('...init class object:', list(Class.__dict__.keys())) class Spam(Eggs, metaclass=SubMetaObj()):# Invoke Sub instance via Super.__call__...rest of file unchanged...c:\code>py −3 metaclass4-super.pymaking class In SuperMetaObj.call:...as before...In SubMetaObj.new:...as before...In SubMetaObj.init:...as before...making instance data: 1 3
# Classes can catch calls too (but built-ins look in metas, not supers!)class SuperMeta(type): def __call__(meta, classname, supers, classdict): print('In SuperMeta.call: ', classname, supers, classdict, sep='\n...') return type.__call__(meta, classname, supers, classdict) def __init__(Class, classname, supers, classdict): print('In SuperMeta init:', classname, supers, classdict, sep='\n...') print('...init class object:', list(Class.__dict__.keys())) print('making metaclass') class SubMeta(type, metaclass=SuperMeta): def __new__(meta, classname, supers, classdict): print('In SubMeta.new: ', classname, supers, classdict, sep='\n...') return type.__new__(meta, classname, supers, classdict) def __init__(Class, classname, supers, classdict): print('In SubMeta init:', classname, supers, classdict, sep='\n...') print('...init class object:', list(Class.__dict__.keys())) class Eggs: pass print('making class') class Spam(Eggs, metaclass=SubMeta):# Invoke SubMeta, via SuperMeta.__call__data = 1 def meth(self, arg): return self.data + arg print('making instance') X = Spam() print('data:', X.data, X.meth(2))
c:\code> py −3 metaclass5.py
making metaclass
In SuperMeta init:
...SubMeta
...(<class 'type'>,)
...{'__init__': <function SubMeta.__init__ at 0x028F92F0>, ...}
...init class object: ['__doc__', '__module__', '__new__', '__init__, ...]
making class
In SuperMeta.call:
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x028F9378>, '__module__': '__main__'}
In SubMeta.new:
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x028F9378>, '__module__': '__main__'}
In SubMeta init:
...Spam
...(<class '__main__.Eggs'>,)
...{'data': 1, 'meth': <function Spam.meth at 0x028F9378>, '__module__': '__main__'}
...init class object: ['__qualname__', '__module__', '__doc__', 'data', 'meth']
making instance
data: 1 3class SuperMeta(type):
def __call__(meta, classname, supers, classdict): # By name, not built-in
print('In SuperMeta.call:', classname)
return type.__call__(meta, classname, supers, classdict)
class SubMeta(SuperMeta): # Created by type default
def __init__(Class, classname, supers, classdict): # Overrides type.__init__
print('In SubMeta init:', classname)
print(SubMeta.__class__)
print([n.__name__ for n in SubMeta.__mro__])
print()
print(SubMeta.__call__) # Not a data descriptor if found by name
print()
SubMeta.__call__(SubMeta, 'xxx', (), {}) # Explicit calls work: class inheritance
print()
SubMeta('yyy', (), {}) # But implicit built-in calls do not: type
c:\code> py −3 metaclass5b.py
<class 'type'>
['SubMeta', 'SuperMeta', 'type', 'object']
<function SuperMeta.__call__ at 0x029B9158>
In SuperMeta.call: xxx
In SubMeta init: xxx
In SubMeta init: yyytype class (usually)Although they have a special role, metaclasses are coded withclassstatements and follow the usual OOP model in Python. For example, as subclasses oftype, they can redefine the type object’s methods, overriding and customizing them as needed. Metaclasses typically redefine thetypeclass’s__new__and__init__to customize class creation and initialization. Although it’s less common, they can also redefine__call__if they wish to catch the end-of-class creation call directly (albeit with the complexities we saw in the prior section), and can even be simple functions or other callables that return arbitrary objects, instead oftypesubclasses.
Themetaclass=Mdeclaration in a user-defined class is inherited by the class’s normal subclasses, too, so the metaclass will run for the construction of each class that inherits this specification in a superclass inheritance chain.
Metaclass declarations specify an instance relationship, which is not the same as what we’ve called inheritance thus far. Because classes are instances of metaclasses, the behavior defined in a metaclass applies to the class, but not the class’s later instances. Instances obtain behavior from their classes and superclasses, but not from any metaclasses. Technically, attribute inheritance for normal instances usually searches only the__dict__dictionaries of the instance, its class, and all its superclasses; metaclasses are not included in inheritance lookup for normal instances.
By contrast, classes do acquire methods of their metaclasses by virtue of the instance relationship. This is a source of class behavior that processes classes themselves. Technically, classes acquire metaclass attributes through the class’s__class__link just as normal instances acquire names from their class, but inheritance via__dict__search is attempted first: when the same name is available to a class in both a metaclass and a superclass, the superclass (inheritance) version is used instead of that on a metaclass (instance). The class’s__class__, however, is not followed for its own instances: metaclass attributes are made available to their instance classes, but not to instances of those instance classes (and see the earlier reference to Dr. Seuss...).
# File metainstance.pyclass MetaOne(type): def __new__(meta, classname, supers, classdict):# Redefine type methodprint('In MetaOne.new:', classname) return type.__new__(meta, classname, supers, classdict) def toast(self): return 'toast' class Super(metaclass=MetaOne):# Metaclass inherited by subs toodef spam(self):# MetaOne run twice for two classesreturn 'spam' class Sub(Super):# Superclass: inheritance versus instancedef eggs(self):# Classes inherit from superclassesreturn 'eggs'# But not from metaclasses
>>>from metainstance import *# Runs class statements: metaclass run twiceIn MetaOne.new: Super In MetaOne.new: Sub >>>X = Sub()# Normal instance of user-defined class>>>X.eggs()# Inherited from Sub'eggs' >>>X.spam()# Inherited from Super 'spam' >>>X.toast()# Not inherited from metaclassAttributeError: 'Sub' object has no attribute 'toast'
>>>Sub.eggs(X)# Own method'eggs' >>>Sub.spam(X)# Inherited from Super'spam' >>>Sub.toast()# Acquired from metaclass'toast' >>>Sub.toast(X)# Not a normal class methodTypeError: toast() takes 1 positional argument but 2 were given
>>>Sub.toast<bound method MetaOne.toast of <class 'metainstance.Sub'>> >>>Sub.spam<function Super.spam at 0x0298A2F0> >>>X.spam<bound method Sub.spam of <metainstance.Sub object at 0x02987438>>
>>>class A(type): attr = 1>>>class B(metaclass=A): pass# B is meta instance and acquires meta attr>>>I = B()# I inherits from class but not meta!>>>B.attr1 >>>I.attrAttributeError: 'B' object has no attribute 'attr' >>>'attr' in B.__dict__, 'attr' in A.__dict__(False, True)
>>>class A: attr = 1>>>class B(A): pass# I inherits from class and supers>>>I = B()>>>B.attr1 >>>I.attr1 >>>'attr' in B.__dict__, 'attr' in A.__dict__(False, True)
>>>class M(type): attr = 1>>>class A: attr = 2>>>class B(A, metaclass=M): pass# Supers have precedence over metas>>>I = B()>>>B.attr, I.attr(2, 2) >>>'attr' in B.__dict__, 'attr' in A.__dict__, 'attr' in M.__dict__(False, True, True)
>>>class M(type): attr = 1>>>class A: attr = 2>>>class B(A): pass>>>class C(B, metaclass=M): pass# Super two levels above meta: still wins>>>I = C()>>>I.attr, C.attr(2, 2) >>>[x.__name__ for x in C.__mro__]# See Chapter 32 for all things MRO['C', 'B', 'A', 'object']
>>>I.__class__# Followed by inheritance: instance's class<class '__main__.C'> >>>C.__bases__# Followed by inheritance: class's supers(<class '__main__.B'>,) >>>C.__class__# Followed by instance acquisition: metaclass<class '__main__.M'> >>>C.__class__.attr# Another way to get to metaclass attributes1
>>>class M1(type): attr1 = 1# Metaclass inheritance tree>>>class M2(M1): attr2 = 2# Gets __bases__, __class__, __mro__>>>class C1: attr3 = 3# Superclass inheritance tree>>>class C2(C1,metaclass=M2): attr4 = 4# Gets __bases__, __class__, __mro__>>>I = C2()# I gets __class__ but not others>>>I.attr3, I.attr4# Instance inherits from super tree(3, 4) >>>C2.attr1, C2.attr2, C2.attr3, C2.attr4# Class gets names from both trees!(1, 2, 3, 4) >>>M2.attr1, M2.attr2# Metaclass inherits names too!(1, 2)
>>>I.__class__# Links followed at instance with no __bases__<class '__main__.C2'> >>>C2.__bases__(<class '__main__.C1'>,) >>>C2.__class__# Links followed at class after __bases__<class '__main__.M2'> >>>M2.__bases__(<class '__main__.M1'>,) >>>I.__class__.attr1# Route inheritance to the class's meta tree1 >>>I.attr1# Though class's __class__ not followed normallyAttributeError: 'C2' object has no attribute 'attr1' >>>M2.__class__# Both trees have MROs and instance links<class 'type'> >>>[x.__name__ for x in C2.__mro__]# __bases__ tree from I.__class__['C2', 'C1', 'object'] >>>[x.__name__ for x in M2.__mro__]# __bases__ tree from C2.__class__['M2', 'M1', 'type', 'object']
__dict__ of the
instance I__dict__ of all
classes on the __mro__
found at I’s __class__,
from left to right__dict__ of all
classes on the __mro__
found at C itself, from left to right__dict__ of all
metaclasses on the __mro__
found at C’s __class__,
from left to right>>>class C: pass# Inheritance special case #1...>>>I = C()# Class data descriptors have precedence>>>I.__class__, I.__dict__(<class '__main__.C'>, {}) >>>I.__dict__['name'] = 'bob'# Dynamic data in the instance>>>I.__dict__['__class__'] = 'spam'# Assign keys, not attributes>>>I.__dict__['__dict__'] = {}>>>I.name# I.name comes from I.__dict__ as usual'bob'# But I.__class__ and I.__dict__ do not!>>>I.__class__, I.__dict__(<class '__main__.C'>, {'__class__': 'spam', '__dict__': {}, 'name': 'bob'})
>>>class D:def __get__(self, instance, owner): print('__get__')def __set__(self, instance, value): print('__set__')>>>class C: d = D()# Data descriptor attribute>>>I = C()>>>I.d# Inherited data descriptor access__get__ >>>I.d = 1__set__ >>>I.__dict__['d'] = 'spam'# Define same name in instance namespace dict>>>I.d# But doesn't hide data descriptor in class!__get__
>>>class D:def __get__(self, instance, owner): print('__get__')>>>class C: d = D()>>>I = C()>>>I.d# Inherited nondata descriptor access__get__ >>>I.__dict__['d'] = 'spam'# Hides class names per normal inheritance rules>>>I.d'spam'
__dict__
of all classes on the __mro__ found at I’s __class____get__
and exit__dict__ of the instance I__dict__
of all metaclasses on the __mro__ found at C’s __class____get__
and exit__dict__ of a class on C’s
own __mro____set__
instead of __get__, and step c stops and stores in the instance
instead of attempting a fetch.>>>class C:# Inheritance special case #2...attr = 1# Built-ins skip a stepdef __str__(self): return('class')>>>I = C()>>>I.__str__(), str(I)# Both from class if not in instance('class', 'class') >>>I.__str__ = lambda: 'instance'>>>I.__str__(), str(I)# Explicit=>instance, built-in=>class!('instance', 'class') >>>I.attr# Asymmetric with normal or explicit names1 >>>I.attr = 2; I.attr2
>>>class D(type):def __str__(self): return('D class')>>>class C(D):pass>>>C.__str__(C), str(C)# Explicit=>super, built-in=>metaclass!('D class', "<class '__main__.C'>") >>>class C(D):def __str__(self): return('C class')>>>C.__str__(C), str(C)# Explicit=>class, built-in=>metaclass!('C class', "<class '__main__.C'>") >>>class C(metaclass=D):def __str__(self): return('C class')>>>C.__str__(C), str(C)# Built-in=>user-defined metaclass('C class', 'D class')
>>>class C(metaclass=D):pass>>>C.__str__(C), str(C)# Explicit=>object, built-in=>metaclass("<class '__main__.C'>", 'D class') >>>C.__str__<slot wrapper '__str__' of 'object' objects> >>>for k in (C, C.__class__, type): print([x.__name__ for x in k.__mro__])['C', 'object'] ['D', 'type', 'object'] ['type', 'object']
Special cases aren’t special enough to break the rules.
>>>class A(type):def x(cls): print('ax', cls)# A metaclass (instances=classes)def y(cls): print('ay', cls)# y is overridden by instance B>>>class B(metaclass=A):def y(self): print('by', self)# A normal class (normal instances)def z(self): print('bz', self)# Namespace dict holds y and z>>>B.x# x acquired from metaclass<bound method A.x of <class '__main__.B'>> >>>B.y# y and z defined in class itself<function B.y at 0x0295F1E0> >>>B.z<function B.z at 0x0295F378> >>>B.x()# Metaclass method call: gets clsax <class '__main__.B'> >>>I = B()# Instance method calls: get inst>>>I.y()by <__main__.B object at 0x02963BE0> >>>I.z()bz <__main__.B object at 0x02963BE0> >>>I.x()# Instance doesn't see meta namesAttributeError: 'B' object has no attribute 'x'
>>>class A(type):def a(cls):# Metaclass method: gets classcls.x = cls.y + cls.z>>>class B(metaclass=A):y, z = 11, 22@classmethod# Class method: gets classdef b(cls):return cls.x>>>B.a()# Call metaclass method; visible to class only>>>B.x# Creates class data on B, accessible to normal instances33 >>>I = B()>>>I.x, I.y, I.z(33, 11, 22) >>>I.b()# Class method: sends class, not instance; visible to instance33 >>>I.a()# Metaclass methods: accessible through class onlyAttributeError: 'B' object has no attribute 'a'
>>>class A(type):def __getitem__(cls, i):# Meta method for processing classes:return cls.data[i]# Built-ins skip class, use meta# Explicit names search class + meta>>>class B(metaclass=A):# Data descriptors in meta used firstdata = 'spam'>>>B[0]# Metaclass instance names: visible to class only's' >>>B.__getitem__<bound method A.__getitem__ of <class '__main__.B'>> >>>I = B()>>>I.data, B.data# Normal inheritance names: visible to instance and class('spam', 'spam') >>>I[0]TypeError: 'B' object does not support indexing
>>>class A(type):def __getattr__(cls, name):# Acquired by class B getitemreturn getattr(cls.data, name)# But not run same by built-ins>>>class B(metaclass=A):data = 'spam'>>>B.upper()'SPAM' >>>B.upper<built-in method upper of str object at 0x029E7420> >>>B.__getattr__<bound method A.__getattr__ of <class '__main__.B'>> >>>I = B()>>>I.upperAttributeError: 'B' object has no attribute 'upper' >>>I.__getattr__AttributeError: 'B' object has no attribute '__getattr__'
>>>B.data = [1, 2, 3]>>>B.append(4)# Explicit normal names routed to meta's getattr>>>B.data[1, 2, 3, 4] >>>B.__getitem__(0)# Explicit special names routed to meta's gettarr1 >>>B[0]# But built-ins skip meta's gettatr too?!TypeError: 'A' object does not support indexing
# Extend manually - adding new methods to classes
class Client1:
def __init__(self, value):
self.value = value
def spam(self):
return self.value * 2
class Client2:
value = 'ni?'
def eggsfunc(obj):
return obj.value * 4
def hamfunc(obj, value):
return value + 'ham'
Client1.eggs = eggsfunc
Client1.ham = hamfunc
Client2.eggs = eggsfunc
Client2.ham = hamfunc
X = Client1('Ni!')
print(X.spam())
print(X.eggs())
print(X.ham('bacon'))
Y = Client2()
print(Y.eggs())
print(Y.ham('bacon'))c:\code> py −3 extend-manual.py
Ni!Ni!
Ni!Ni!Ni!Ni!
baconham
ni?ni?ni?ni?
baconham# Extend with a metaclass - supports future changes better
def eggsfunc(obj):
return obj.value * 4
def hamfunc(obj, value):
return value + 'ham'
class Extender(type):
def __new__(meta, classname, supers, classdict):
classdict['eggs'] = eggsfunc
classdict['ham'] = hamfunc
return type.__new__(meta, classname, supers, classdict)
class Client1(metaclass=Extender):
def __init__(self, value):
self.value = value
def spam(self):
return self.value * 2
class Client2(metaclass=Extender):
value = 'ni?'
X = Client1('Ni!')
print(X.spam())
print(X.eggs())
print(X.ham('bacon'))
Y = Client2()
print(Y.eggs())
print(Y.ham('bacon'))c:\code> py −3 extend-meta.py
Ni!Ni!
Ni!Ni!Ni!Ni!
baconham
ni?ni?ni?ni?
baconham# Can also configure class based on runtime tests
class MetaExtend(type):
def __new__(meta, classname, supers, classdict):
if sometest():
classdict['eggs'] = eggsfunc1
else:
classdict['eggs'] = eggsfunc2
if someothertest():
classdict['ham'] = hamfunc
else:
classdict['ham'] = lambda *args: 'Not supported'
return type.__new__(meta, classname, supers, classdict)class statement, after the new class has
been created.class statement, in order to create the
new class.# Extend with a decorator: same as providing __init__ in a metaclassdef eggsfunc(obj): return obj.value * 4 def hamfunc(obj, value): return value + 'ham' def Extender(aClass): aClass.eggs = eggsfunc# Manages class, not instanceaClass.ham = hamfunc# Equiv to metaclass __init__return aClass @Extender class Client1:# Client1 = Extender(Client1)def __init__(self, value):# Rebound at end of class stmtself.value = value def spam(self): return self.value * 2 @Extender class Client2: value = 'ni?' X = Client1('Ni!')# X is a Client1 instanceprint(X.spam()) print(X.eggs()) print(X.ham('bacon')) Y = Client2() print(Y.eggs()) print(Y.ham('bacon'))
# Class decorator to trace external instance attribute fetchesdef Tracer(aClass):# On @ decoratorclass Wrapper: def __init__(self, *args, **kargs):# On instance creationself.wrapped = aClass(*args, **kargs)# Use enclosing scope namedef __getattr__(self, attrname): print('Trace:', attrname)# Catches all but .wrappedreturn getattr(self.wrapped, attrname)# Delegate to wrapped objectreturn Wrapper @Tracer class Person:# Person = Tracer(Person)def __init__(self, name, hours, rate):# Wrapper remembers Personself.name = name self.hours = hours self.rate = rate# In-method fetch not traceddef pay(self): return self.hours * self.rate bob = Person('Bob', 40, 50)# bob is really a Wrapperprint(bob.name)# Wrapper embeds a Personprint(bob.pay())# Triggers __getattr__
c:\code> py −3 manage-inst-deco.py
Trace: name
Bob
Trace: pay
2000# Manage instances like the prior example, but with a metaclassdef Tracer(classname, supers, classdict):# On class creation callaClass = type(classname, supers, classdict)# Make client classclass Wrapper: def __init__(self, *args, **kargs):# On instance creationself.wrapped = aClass(*args, **kargs) def __getattr__(self, attrname): print('Trace:', attrname)# Catches all but .wrappedreturn getattr(self.wrapped, attrname)# Delegate to wrapped objectreturn Wrapper class Person(metaclass=Tracer):# Make Person with Tracerdef __init__(self, name, hours, rate):# Wrapper remembers Personself.name = name self.hours = hours self.rate = rate# In-method fetch not traceddef pay(self): return self.hours * self.rate bob = Person('Bob', 40, 50)# bob is really a Wrapperprint(bob.name)# Wrapper embeds a Personprint(bob.pay())# Triggers __getattr__
c:\code> py −3 manage-inst-meta.py
Trace: name
Bob
Trace: pay
2000# A decorator can call a metaclass, though not vice versa without type()>>>class Metaclass(type):def __new__(meta, clsname, supers, attrdict):print('In M.__new__:')print([clsname, supers, list(attrdict.keys())])return type.__new__(meta, clsname, supers, attrdict)>>>def decorator(cls):return Metaclass(cls.__name__, cls.__bases__, dict(cls.__dict__))>>>class A:x = 1>>>@decoratorclass B(A):y = 2def m(self): return self.x + self.yIn M.__new__: ['B', (<class '__main__.A'>,), ['__qualname__', '__doc__', 'm', 'y', '__module__']] >>>B.x, B.y(1, 2) >>>I = B()>>>I.x, I.y, I.m()(1, 2, 3)
>>>class B(A, metaclass=Metaclass): ...# Same effect, but makes just one class
>>>def Metaclass(clsname, supers, attrdict):return decorator(type(clsname, supers, attrdict))>>>def decorator(cls): ...>>>class B(A, metaclass=Metaclass): ...# Metas can call decos and vice versa
>>>def func(name, supers, attrs):return 'spam'>>>class C(metaclass=func):# A class whose metaclass makes it a string!attr = 'huh?'>>>C, C.upper()('spam', 'SPAM') >>>def func(cls):return 'spam'>>>@funcclass C:# A class whose decorator makes it a string!attr = 'huh?'>>>C, C.upper()('spam', 'SPAM')
# File decotools.py: assorted decorator toolsimport time def tracer(func):# Use function, not class with __call__calls = 0# Else self is decorator instance onlydef onCall(*args, **kwargs): nonlocal calls calls += 1 print('call %s to %s' % (calls, func.__name__)) return func(*args, **kwargs) return onCall def timer(label='', trace=True):# On decorator args: retain argsdef onDecorator(func):# On @: retain decorated funcdef onCall(*args, **kargs):# On calls: call originalstart = time.clock()# State is scopes + func attrresult = func(*args, **kargs) elapsed = time.clock() - start onCall.alltime += elapsed if trace: format = '%s%s: %.5f, %.5f' values = (label, func.__name__, elapsed, onCall.alltime) print(format % values) return result onCall.alltime = 0 return onCall return onDecorator
from decotools import tracer
class Person:
@tracer
def __init__(self, name, pay):
self.name = name
self.pay = pay
@tracer
def giveRaise(self, percent): # giveRaise = tracer(giverRaise)
self.pay *= (1.0 + percent) # onCall remembers giveRaise
@tracer
def lastName(self): # lastName = tracer(lastName)
return self.name.split()[-1]
bob = Person('Bob Smith', 50000)
sue = Person('Sue Jones', 100000)
print(bob.name, sue.name)
sue.giveRaise(.10) # Runs onCall(sue, .10)
print('%.2f' % sue.pay)
print(bob.lastName(), sue.lastName()) # Runs onCall(bob), remembers lastNamec:\code> py −3 decoall-manual.py
call 1 to __init__
call 2 to __init__
Bob Smith Sue Jones
call 1 to giveRaise
110000.00
call 1 to lastName
call 2 to lastName
Smith Jones# Metaclass that adds tracing decorator to every method of a client classfrom types import FunctionType from decotools import tracer class MetaTrace(type): def __new__(meta, classname, supers, classdict): for attr, attrval in classdict.items(): if type(attrval) is FunctionType:# Method?classdict[attr] = tracer(attrval)# Decorate itreturn type.__new__(meta, classname, supers, classdict)# Make classclass Person(metaclass=MetaTrace): def __init__(self, name, pay): self.name = name self.pay = pay def giveRaise(self, percent): self.pay *= (1.0 + percent) def lastName(self): return self.name.split()[-1] bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) print(bob.name, sue.name) sue.giveRaise(.10) print('%.2f' % sue.pay) print(bob.lastName(), sue.lastName())
c:\code> py −3 decoall-meta.py
call 1 to __init__
call 2 to __init__
Bob Smith Sue Jones
call 1 to giveRaise
110000.00
call 1 to lastName
call 2 to lastName
Smith Jones# Metaclass factory: apply any decorator to all methods of a classfrom types import FunctionType from decotools import tracer, timer def decorateAll(decorator): class MetaDecorate(type): def __new__(meta, classname, supers, classdict): for attr, attrval in classdict.items(): if type(attrval) is FunctionType: classdict[attr] = decorator(attrval) return type.__new__(meta, classname, supers, classdict) return MetaDecorate class Person(metaclass=decorateAll(tracer)):# Apply a decorator to alldef __init__(self, name, pay): self.name = name self.pay = pay def giveRaise(self, percent): self.pay *= (1.0 + percent) def lastName(self): return self.name.split()[-1] bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) print(bob.name, sue.name) sue.giveRaise(.10) print('%.2f' % sue.pay) print(bob.lastName(), sue.lastName())
c:\code> py −3 decoall-meta-any.py
call 1 to __init__
call 2 to __init__
Bob Smith Sue Jones
call 1 to giveRaise
110000.00
call 1 to lastName
call 2 to lastName
Smith Jonesclass Person(metaclass=decorateAll(tracer)):# Apply tracerclass Person(metaclass=decorateAll(timer())):# Apply timer, defaultsclass Person(metaclass=decorateAll(timer(label='**'))):# Decorator arguments
# If using timer: total time per method
print('-'*40)
print('%.5f' % Person.__init__.alltime)
print('%.5f' % Person.giveRaise.alltime)
print('%.5f' % Person.lastName.alltime)c:\code> py −3 decoall-meta-any2.py
**__init__: 0.00001, 0.00001
**__init__: 0.00001, 0.00001
Bob Smith Sue Jones
**giveRaise: 0.00002, 0.00002
110000.00
**lastName: 0.00002, 0.00002
**lastName: 0.00002, 0.00004
Smith Jones
----------------------------------------
0.00001
0.00002
0.00004# Class decorator factory: apply any decorator to all methods of a classfrom types import FunctionType from decotools import tracer, timer def decorateAll(decorator): def DecoDecorate(aClass): for attr, attrval in aClass.__dict__.items(): if type(attrval) is FunctionType: setattr(aClass, attr, decorator(attrval))# Not __dict__return aClass return DecoDecorate @decorateAll(tracer)# Use a class decoratorclass Person:# Applies func decorator to methodsdef __init__(self, name, pay):# Person = decorateAll(..)(Person)self.name = name# Person = DecoDecorate(Person)self.pay = pay def giveRaise(self, percent): self.pay *= (1.0 + percent) def lastName(self): return self.name.split()[-1] bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) print(bob.name, sue.name) sue.giveRaise(.10) print('%.2f' % sue.pay) print(bob.lastName(), sue.lastName())
c:\code> py −3 decoall-deco-any.py
call 1 to __init__
call 2 to __init__
Bob Smith Sue Jones
call 1 to giveRaise
110000.00
call 1 to lastName
call 2 to lastName
Smith Jones@decorateAll(tracer)# Decorate all with tracer@decorateAll(timer())# Decorate all with timer, defaults@decorateAll(timer(label='@@'))# Same but pass a decorator argument
# If using timer: total time per method
print('-'*40)
print('%.5f' % Person.__init__.alltime)
print('%.5f' % Person.giveRaise.alltime)
print('%.5f' % Person.lastName.alltime)c:\code> py −3 decoall-deco-any2.py
@@__init__: 0.00001, 0.00001
@@__init__: 0.00001, 0.00001
Bob Smith Sue Jones
@@giveRaise: 0.00002, 0.00002
110000.00
@@lastName: 0.00002, 0.00002
@@lastName: 0.00002, 0.00004
Smith Jones
----------------------------------------
0.00001
0.00002
0.00004@decorateAll(tracer(timer(label='@@')))# Traces applying the timerclass Person: @decorateAll(tracer)# Traces onCall wrapper, times methods@decorateAll(timer(label='@@')) class Person: @decorateAll(timer(label='@@')) @decorateAll(tracer)# Times onCall wrapper, traces methodsclass Person:
type
class by default. Metaclasses are usually subclasses of the type class, which redefines class creation
protocol methods in order to customize the class creation call issued
at the end of a class statement;
they typically redefine the methods __new__ and __init__ to tap into the class creation
protocol. Metaclasses can also be coded other ways — as simple
functions, for example — but they are always responsible for making and
returning an object for the new class. Metaclasses may have methods
and data to provide behavior for their classes too — and constitute a
secondary pathway for inheritance search — but their attributes are
accessible only to their class instances, not to their instance’s
instances.class header line: class C(metaclass=M). In Python 2.X, use a class attribute
instead: __metaclass__ =
M. In 3.X, the class header line can also name normal
superclasses before the metaclass
keyword argument; in 2.X you generally should derive from object too, though this is sometimes
optional.class statement, class decorators
and metaclasses can both be used to manage classes. Decorators rebind
a class name to a callable’s result and metaclasses route class
creation through a callable, but both hooks can be used for similar
purposes. To manage classes, decorators simply augment and return the
original class objects. Metaclasses augment a class after they create
it. Decorators may have a slight disadvantage in this role if a new
class must be defined, because the original class has already been
created.class statement, we can use both
class decorators and metaclasses to manage class instances, by
inserting a wrapper (proxy) object to catch instance creation calls.
Decorators may rebind the class name to a callable run on instance
creation that retains the original class object. Metaclasses can do
the same, but may have a slight disadvantage in this role, because
they must also create the class object.1 And to quote a Python 3.3 error message I just came across: “TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases” (!). This reflects an erroneous use of a module as a superclass, but metaclasses may not be as optional as developers imply — a theme we’ll revisit in the next chapter’s conclusion to this book.
Generators, decorators, slots, properties, descriptors, metaclasses, context managers, closures,super, namespace packages, Unicode, function annotations, relative imports, keyword-only arguments, class and static methods, and even obscure applications of comprehensions and operator overloading
| Category | Specifics |
|---|---|
#!/usr/bin/python """ File certificate.py: a Python 2.X and 3.X script. Generate a bare-bones class completion certificate: printed, and saved in text and html files displayed in a web browser. """ from __future__ import print_function# 2.X compatibilityimport time, sys, webbrowser if sys.version_info[0] == 2:# 2.X compatibilityinput = raw_input import cgi htmlescape = cgi.escape else: import html htmlescape = html.escape maxline = 60# For seperator linesbrowser = True# Display in a browsersaveto = 'Certificate.txt'# Output filenamestemplate = """ %s ===> Official Certificate <=== Date: %s This certifies that: \t%s has survived the massive tome: \t%s and is now entitled to all privileges thereof, including the right to proceed on to learning how to develop Web sites, desktop GUIs, scientific models, and assorted apps, with the possible assistance of follow-up applications books such as Programming Python (shameless plug intended). --Mark Lutz, Instructor (Note: certificate void where obtained by skipping ahead.) %s """# Interact, setupfor c in 'Congratulations!'.upper(): print(c, end=' ') sys.stdout.flush()# Else some shells wait for \ntime.sleep(0.25) print() date = time.asctime() name = input('Enter your name: ').strip() or 'An unknown reader' sept = '*' * maxline book = 'Learning Python 5th Edition'# Make text file versionfile = open(saveto, 'w') text = template % (sept, date, name, book, sept) print(text, file=file) file.close()# Make html file versionhtmlto = saveto.replace('.txt', '.html') file = open(htmlto, 'w') tags = text.replace(sept, '<hr>')# Insert a few tagstags = tags.replace('===>', '<h1 align=center>') tags = tags.replace('<===', '</h1>') tags = tags.split('\n')# Line-by-line modstags = ['<p>' if line == '' else line for line in tags] tags = ['<i>%s</i>' % htmlescape(line) if line[:1] == '\t' else line for line in tags] tags = '\n'.join(tags) link = '<i><a href="http://learning-python.com/books">Book support site</a></i>\n' foot = '<table>\n<td><img src="ora-lp.jpg" hspace=5>\n<td>%s</table>\n' % link tags = '<html><body bgcolor=beige>' + tags + foot + '</body></html>' print(tags, file=file) file.close()# Display resultsprint('[File: %s]' % saveto, end='') print('\n' * 2, open(saveto).read()) if browser: webbrowser.open(saveto, new=True) webbrowser.open(htmlto, new=False) if sys.platform.startswith('win'): input('[Press Enter]')# Keep window open if clicked on Windows
python at the prompt. Python, IDLE, and
its tkinter GUI toolkit are standard components of this
system.python at a shell prompt
(a.k.a. terminal window), and see what happens. Alternatively, try
searching for “python” in the usual places — /usr/bin, /usr/local/bin, etc. As on Macs, Python
is a standard part of Linux systems.A package that combines Python with extensions for scientific, Windows, and other development needs, including PyWin32 and the PythonWin IDE
A combination of Python and a host of additional libraries and tools oriented toward scientific computing needs
A blend of Python and add-on packages configured to run directly from a portable device
A scientific-oriented Python distribution based on Qt and Spyder
A bundle targeted at business, desktop, and database applications
A commercial distribution for numerical analysis
A distribution for analysis and visualization of large data sets
For Windows (including XP, Vista, 7, and 8), Python comes as a self-installer MSI program file — simply double-click on its file icon, and answer Yes or Next at every prompt to perform a default install. The default install includes Python’s documentation set and support fortkinter(Tkinterin Python 2.X) GUIs, shelve databases, and the IDLE development GUI. Python 3.3 and 2.7 are normally installed in the directories C:\Python33 and C:\Python27 though this can be changed at install time.For convenience, on Windows 7 and earlier Python shows up after the install in the Start button’s All Programs menu (see ahead for Windows 8 notes). Python’s menu there has five entries that give quick access to common tasks: starting the IDLE user interface, reading module documentation, starting an interactive session, reading Python’s standard manuals, and uninstalling. Most of these options involve concepts explored in detail elsewhere in this text.When installed on Windows, Python also automatically uses filename associations to register itself to be the program that opens Python files when their icons are clicked (a program launch technique described in Chapter 3). It is also possible to build Python from its source code on Windows, but this is not commonly done so we’ll skip the details here (see python.org).Three additional install-related notes for Windows users: first, be sure to see the next appendix for an introduction to the new Windows launcher shipped with 3.3; it changes some of the rules for installation, file associations, and command lines, but can be an asset if you have multiple Python versions on your computer (e.g., both 2.X and 3.X). Per Appendix B, Python 3.3’s MSI installer also has an option to set your PATH variable to include Python’s directory.Second, Windows 8 users should see the sidebar in this appendix “Using Python on Windows 8”. Standard Python installs and works the same on Windows 8, where it runs in desktop mode, but you won’t get the Start button menu described earlier, and the tablet interface on top is not yet directly supported.Finally, some Windows Vista users may run into install issues related to security features. This seems to have been resolved over time (and Vista is relatively rare these days), but if running the MSI installer file directly doesn’t work as expected, it’s probably because MSI files are not true executables and do not correctly inherit administrator permissions (they run per the registry). To fix, run the installer from a command line with appropriate permissions: Select Command Prompt, choose “Run as administrator,” cd to the directory where your Python MSI file resides, and run the MSI installer with a command line of the form:msiexec /i python-2.5.1.msi.
For Linux, if Python or your desired flavor of it is not already present, you can probably obtain it as one or more RPM files, which you unpack in the usual way (consult the RPM manpage for details). Depending on which RPMs you download, there may be one for Python itself, and another that adds support fortkinterGUIs and the IDLE environment. Because Linux is a Unix-like system, the next paragraph applies as well.
For Unix systems, Python is usually compiled from its full C source code distribution. This usually only requires you to unpack the file and run simpleconfigureandmakecommands; Python configures its own build procedure automatically, according to the system on which it is being compiled. However, be sure to see the package’s README file for more details on this process. Because Python is open source, its source code may be used and distributed free of charge.
| Variable | Role |
|---|---|
PATHThePATHsetting lists a set of directories that the operating system searches for executable programs, when they are invoked without a full directory path. It should normally include the directory where your Python interpreter lives (the python program on Unix, or the python.exe file on Windows).You don’t need to set this variable at all if you are willing to work in the directory where Python resides, or type the full path to Python in command lines. On Windows, for instance, thePATHis irrelevant if you run acd C:\Python33before running any code (to change to the directory where Python lives — though you shouldn’t generally store your own code in this directory per Chapter 3), or always typeC:\Python33\pythoninstead of justpython(giving a full path).Also note thatPATHsettings are mostly for launching programs from command lines; they are usually irrelevant when launching via icon clicks and IDEs — the former uses filename associations, and the latter uses built-in mechanisms, and doesn’t generally require this configuration step. See also Appendix B for details on 3.3’s automaticPATHsetting option at install time.
PYTHONPATHThePYTHONPATHsetting serves a role similar toPATH: the Python interpreter consults thePYTHONPATHvariable to locate module files when you import them in a program. If used, this variable is set to a platform-dependent list of directory names, separated by colons on Unix and semicolons on Windows. This list normally includes just your own source code directories. Its content is merged into thesys.pathmodule import search path, along with the script’s container directory, any .pth path file settings, and standard library directories.You don’t need to set this variable unless you will be performing cross-directory imports — because Python always searches the home directory of the program’s top-level file automatically, this setting is required only if a module needs to import another module that lives in a different directory. See also the discussion of .pth path files later in this appendix for an alternative toPYTHONPATH. For more on the module search path, refer to Chapter 22.
PYTHONSTARTUPIfPYTHONSTARTUPis set to the pathname of a file of Python code, Python executes the file’s code automatically whenever you start the interactive interpreter, as though you had typed it at the interactive command line. This is a rarely used but handy way to make sure you always load certain utilities when working interactively; it saves an import each time you start a Python session.
tkinter settingsIf you wish to use thetkinterGUI toolkit (namedTkinterin 2.X), you might have to set the two GUI variables in the last line of Table A-1 to the names of the source library directories of the Tcl and Tk systems (much likePYTHONPATH). However, these settings are not required on Windows systems (wheretkintersupport is installed alongside Python), and are usually not required on Mac OS X and Linux systems, unless the underlying Tcl and Tk libraries are either invalid or reside in nonstandard directories (see python.org’s Download page for more details).
PY_PYTHON, PY_PYTHON3, PY_PYTHON2These settings are used to specify default Pythons when you are using the new (at this writing) Windows launcher that ships with Python 3.3 and is available separately for other versions. Since we’ll be exploring the launcher in Appendix B, I’ll postpone further details here.
import spam
setenv PYTHONPATH /usr/home/pycode/utilities:/usr/lib/pycode/package1
export PYTHONPATH="/usr/home/pycode/utilities:/usr/lib/pycode/package1"
set PYTHONPATH=c:\pycode\utilities;d:\pycode\package1
c:\pycode\utilities d:\pycode\package1
python [-bBdEhiOqsSuvVWxX] [-ccommand| -mmodule-name|script| - ] [args]
C:\code> python -h# File showargs.py
import sys
print(sys.argv)C:\code>python showargs.py a b -c# Most common: run a script file['showargs.py', 'a', 'b', '-c']
C:\code>python -c "print(2 ** 100)"# Read code from command argument1267650600228229401496703205376 C:\code>python -c "import showargs"# Import a file to run its code['-c'] C:\code>python - < showargs.py a b -c# Read code from standard input['-', 'a', 'b', '-c'] C:\code>python - a b -c < showargs.py# Same effect as prior line['-', 'a', 'b', '-c']
C:\code>python -m showargs a b -c# Locate/run module as script['c:\\code\\showargs.py', 'a', 'b', '-c']
C:\code>python# Interactive debugger session>>>import pdb>>>pdb.run('import showargs')...more omitted: see pdb docsC:\code>python -m pdb showargs.py a b -c# Debugging a script (c=continue)> C:\code\showargs.py(2)<module>() -> import sys (Pdb)c['showargs.py', 'a', 'b', '-c'] ...more omitted: q to exit
C:\code>python -m profile showargs.py a b -c# Profiling a script['showargs.py', 'a', 'b', '-c'] 9 function calls in 0.016 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 2 0.000 0.000 0.000 0.000 :0(charmap_encode) 1 0.000 0.000 0.000 0.000 :0(exec) ...more omitted: see profile docs
c:\code>python -m idlelib.idle -n# Run IDLE in package, no subprocessc:\code>python -m pydoc -b# Run pydoc and timeit tools modulesc:\code>python -m timeit -n 1000 -r 3 -s "L = [1,2,3,4,5]" "M = [x + 1 for x in L]"
C:\code>python -O showargs.py a b -c# Optimized: make/run ".pyo" byte codeC:\code>python -u showargs.py a b -c# Unbuffered standard output stream
C:\code>python -i showargs.py a b -c# Go to interactive mode on script exit['showargs.py', 'a', 'b', '-c'] >>>sys# Final value of sys: imported module<module 'sys' (built-in)> >>>^Z
C:\code>type divbad.pyX = 0 print(1 / X) C:\code>python divbad.py# Run the buggy script...error text omittedZeroDivisionError: division by zero C:\code>python -i divbad.py# Print variable values at error...error text omittedZeroDivisionError: division by zero >>>X0 >>>import pdb# Start full debugger session now>>>pdb.pm()> C:\code\divbad.py(2)<module>() -> print(1 / X) (Pdb)quit
C:\code> c:\python27\python -hC:\code>py what.py# Windows launcher command lines3.3.0 C:\code>py −2 what.py# Version number switch2.7.3 C:\code>py −3.3 -i what.py -a -b -c# Arguments for all 3: py, python, script3.3.0 >>> ^Z
1 Lest that seem too sarcastic, I should note that Windows 8.1 may address some launch screen and Start button (if not menu) concerns per late-breaking rumors, and this edition’s new Windows 8 sidebar replaces one in prior editions that discussed a Windows Vista issue. Any similarities you might deduce from that are officially coincidental.
#!/usr/local/bin/python ...script's code# Run under this specific program#!/usr/bin/env python ...script's code# Run under "python" found on PATH
py.exe for console
programspyw.exe for nonconsole
(typically GUI) programsPATH
settings when used as command lines#! comment lines at the top of scripts to
determine which Python version should be used to run a file’s
code#!python2#!/usr/bin/python2.7#!/usr/bin/env
python3py −2 m.pypy −2.7 m.pypy −3 m.py#!python3 ... ...a 3.X script# Runs under latest 3.X installed... #!python2 ... ...a 2.X script# Runs under latest 2.X installed... #!python2.6 ... ...a 2.6 script# Runs under 2.6 (only)...
C:\code>script.py# Run per file's #! line if present, else per defaultC:\code>py script.py# Ditto, but py.exe is run explicitly
C:\code>py −3 script.py# Runs under latest 3.XC:\code>py −2 script.py# Runs under latest 2.XC:\code>py −2.6 script.py# Runs under 2.6 (only)
C:\code>py −3# Starts latest 3.X, interactiveC:\code>py −2# Starts latest 2.X, interactiveC:\code>py −3.1# Starts 3.1 (only), interactiveC:\code>py# Starts default Python (initially 2.X: see ahead)
#! python3.2 ... ...a 3.X script... C\code>py script.py# Runs under 3.2, per file directiveC\code>py −3.1 script.py# Runs under 3.1, even if 3.2 present
#!python3
import sys
print(sys.version.split()[0]) # First part of string#! python3 import sys print(sys.version.split()[0]) C:\code>what.py# Run per file directive3.3.0 C:\code>py what.py# Ditto: latest 3.X3.3.0
#! python2 ...rest of script unchangedC:\code>what.py# Run with latest 2.X per #!2.7.3
#! python3.1 ... C:\code>what.py# Run with 3.1 per #!3.1.4
#! python2.6
...
C:\code> what.py
Requested Python version (2.6) is not installed#!/bin/python ... C:\code>what.pyUnable to create process using '/bin/python "C:\code\what.py" ' C:\code>py what.pyUnable to create process using '/bin/python what.py' C:\code>py −3 what.py3.3.0
#!/usr/bin/env python*#!/usr/bin/python*#!/usr/local/bin/python*#!python*
python3)To run the version installed with the highest minor release number among those with the major release number given
python3.1)To run that specific version only, optionally suffixed by−32to prefer a 32-bit version (e.g.,python3.1-32)
python)To run the launcher’s default version, which is2unless changed (e.g., by setting thePY_PYTHONenvironment variable to3), another pitfall described ahead
#!python3 [any python.exe arguments go here]
...# not a launcher directive ... C:\code>py −3 what.py# Run per command-line switch3.3.0 C:\code>py −2 what.py# Ditto: latest 2.X installed2.7.3 C:\code>py −3.2 what.py# Ditto: 3.2 specifically (and only)3.2.3 C:\code>py what.py# Run per launcher's default (ahead)2.7.3
#! python3.1 ... C:\code>what.py# Run per file directive3.1.4 C:\code>py what.py# Ditto3.1.4 C:\code>py −3.2 what.py# Switches override directives3.2.3 C:\code>py −2 what.py# Ditto2.7.3
−2Launch the latest Python 2.X version-3Launch the latest Python 3.X version-X.YLaunch the specified Python version (X is 2 or 3)-X.Y−32Launch the specified 32-bit Python version
py[py.exe arg][python.exe args]script.py[script.py args]
# args.py, show my arguments too import sys print(sys.version.split()[0]) print(sys.argv) C:\code>py −3 -i args.py -a 1 -b -c# −3: py, -i: python, rest: script3.3.0 ['args.py', '-a', '1', '-b', '-c'] >>>^ZC:\code>py -i args.py -a 1 -b -c# Args to python, script2.7.3 ['args.py', '-a', '1', '-b', '-c'] >>>^ZC:\code>py −3 -c print(99)# −3 to py, rest to python: "-c cmd"99 C:\code>py −2 -c "print 99"99
#!python ...# Same as #!/usr/bin/pythonC:\code>what.py# Run per launcher default2.7.3
# not a launcher directive ... C:\code>what.py# Also run per default2.7.3 C:\code>py what.py# Ditto2.7.3
# not a launcher directive ... C:\code>what.py# Run per default2.7.3 C:\code>set PY_PYTHON=3# Or via Control Panel/SystemC:\code>what.py# Run per changed default3.3.0
#!python3 ... C:\code>py what.py# Runs "latest" 3.X3.3.0 C:\code>set PY_PYTHON3=3.1# Use PY_PYTHON2 for 2.XC:\code>py what.py# Override highest-minor choice3.1.4
#! lines
now make scripts fail on Windows.PATH extension is off
by default and seems contradictory.str strings support all Unicode
text including ASCII, and the separate bytes type represents raw 8-bit byte
sequences. In 2.X, normal str
strings support both 8-bit text including ASCII, and a separate
unicode type represents richer
Unicode text as an option.open are specialized by
content — text files implement Unicode encodings and represent content
as str strings, and binary files
represent content as bytes
strings. In 2.X, files use distinct interfaces — files created by
open represent content as
str strings for content that is
either 8-bit text or bytes-based data, and codecs.open implements Unicode text
encodings.object automatically and
acquire the numerous changes and extensions of
new-style classes, including their differing
inheritance algorithm, built-ins dispatch, and MRO search order for
diamond-pattern trees. In 2.X, normal classes follow the
classic model, and explicit inheritance from
object or other built-in types
enables the new-style model as an option.map, zip, range, filter, and dictionary keys, values, and items are all iterable objects that
generate values on request. In 2.X, these calls create physical
lists.from . relative import
statements, but 3.X changes the search rule to skip a package’s own
directory for normal imports.// floor division operator,
but the / is true division in 3.X
and retains fractional remainders, while / is type-specific in 2.X.int and extended long, and automatic conversion to long.pydoc –b interface is supported as of 3.2
and required as of 3.3. In 2.X, the original pydoc –g GUI client interface may be used
instead.__pycache__
subdirectory of the source directory, with version-identifying
names. In 2.X, byte code is stored in the source file directory with
generic names.key mappers
instead). In 2.X all these forms work.string module functions redundant with
string object methods are also removed in 3.X.reload, apply, `x`, <>, 0177, 999L, dict.has_key,
raw_input, xrange, file, reduce, and file.xreadlines.* in sequence assignment
targets to collect remaining unmatched iterable items in a list. 2.X
can achieve similar effects with slicing.nonlocal statement, which allows names in
enclosing function scopes to be changed from within nested
functions. 2.X can achieve similar effects with function attributes,
mutable objects, and class state.raise from extension; 3.3 allows a
None to cancel the chain.yield statement may delegate to a nested
generator with from. 2.X can
often achieve similar results with a for loop in simpler use cases.{1, 4, 2, 3, 4}{c * 4 for c in 'spam'}, {c: c * 4 for c in 'spam'}dict.viewkeys(), dict.viewvalues(), dict.viewitems()
str.format (from 3.1):'{:,.2f} {}'.format(1234567.891, 'spam')with statement context managers (from
3.1):with X() as x, Y() as y: ...
repr display
improvements (back-ported from 3.1: see ahead)yield from ... (see Chapter 20)raise ... from None (see Chapter 34)u'xxxx' the same as its normal string
'xxxx', similar to the way 2.X
treats 3.X’s bytes literal b'xxxx' the same as its normal string
'xxxx' (see Chapter 4, Chapter 7, and Chapter 37)pydoc
-b, replacing its former standalone GUI client search
interface, which was in the Windows 7 and earlier Start button and
invoked by pydoc –g (see Chapter 15)ftplib, time, and email, and potentially
distutils; impacts in this book: time has new portable calls in 3.X (see Chapter 21 and Chapter 39)__import__ function in importlib.__import__, in part to unify and
more clearly expose its implementation (see Chapter 22 and Chapter 25)PATH setting to include
3.3’s directory as an install-time option to simplify some command
lines (see Appendixes A and B)#! lines for
dispatching Python scripts on Windows, and allows both #! lines and new py command lines to select between Python
2.X and 3.X versions explicitly on both a per-file and per-command
basis (see the new Appendix B)__pycache__ (see Chapter 2 and Chapter 22)struct module’s
autoencoding for strings is gone (see Chapter 9 and Chapter 37)str/bytes split supported better by Python
itself (not relevant to this book)cgi.escape call was to
be moved in 3.2+ (not relevant to this book)format method calls (Chapter 7)with statements (Chapter 34)| Extension | Covered in chapter(s) |
|---|---|
| Removed | Replacement | Covered in chapter(s) |
|---|---|---|
B if A else C
conditional expression (Chapter 12,
Chapter 19)with/as context managers (Chapter 34)try/except/finally unification (Chapter 34)sorted, sum, any,
all, enumerate (Chapter 13 and Chapter 14)distutils, unittest and doctest, IDLE enhancements, Shed Skin, and
so on (Chapter 2 and Chapter 36)%python...copyright information lines...>>>"Hello World!"'Hello World!' >>># Use Ctrl-D or Ctrl-Z to exit, or close window
print('Hello module world!')
% python module1.py
Hello module world!%python>>>import module1Hello module world! >>>
#! trick, your solution will
look like the following (although your #! line may need to list another path on
your machine). Note that these lines are significant under the Windows
launcher shipped and installed with Python 3.3, where they are parsed
to select a version of Python to run the script, along with a default
setting; see Appendix B for
details and examples.#!/usr/local/bin/python(or #!/usr/bin/env python)print('Hello module world!') %chmod +x module1.py%module1.pyHello module world!
try statements and
process them arbitrarily; you’ll also see there that Python includes a
full-blown source code debugger for special error-detection
requirements. For now, notice that Python gives meaningful messages
when programming errors occur, instead of crashing silently:%python>>>2 ** 50032733906078961418700131896968275991522166420460430647894832913680961337964046745 54883270092325904157150886684127560071009217256545885393053328527589376 >>> >>>1 / 0Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: int division or modulo by zero >>> >>>spamTraceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'spam' is not defined
L = [1, 2] L.append(L)
[1, 2, [1, 2, [1, 2, [1, 2, and so on, until
you hit the break-key combination on your machine (which, technically,
raises a keyboard-interrupt exception that prints a default message).
Beginning with Python 1.5.1, the printer is clever enough to detect
cycles and prints [[...]] instead
to let you know that it has detected a loop in the object’s structure
and avoided getting stuck printing forever.L becomes a named
reference to a two-item list object — a pointer to a piece of memory.
Python lists are really arrays of object references, with an append method that changes the array in
place by tacking on another object reference at the end. Here, the
append call adds a reference to the
front of L at the end of L, which leads to the cycle illustrated in
Figure D-1: a pointer
at the end of the list that points back to the front of the
list.; is used in a few of these to squeeze more
than one statement onto a single line (the ; is a statement separator), and commas
build up tuples displayed in parentheses. Also keep in mind that the
/ division result near the top
differs in Python 2.X and 3.X (see Chapter 5
for details), and the list wrapper
around dictionary method calls is needed to display results in 3.X,
but not 2.X (see Chapter 8):# Numbers>>>2 ** 16# 2 raised to the power 1665536 >>>2 / 5, 2 / 5.0# Integer / truncates in 2.X, but not 3.X(0.40000000000000002, 0.40000000000000002)# Strings>>>"spam" + "eggs"# Concatenation'spameggs' >>>S = "ham">>>"eggs " + S'eggs ham' >>>S * 5# Repetition'hamhamhamhamham' >>>S[:0]# An empty slice at the front -- [0:0]''# Empty of same type as object sliced>>>"green %s and %s" % ("eggs", S)# Formatting'green eggs and ham' >>>'green {0} and {1}'.format('eggs', S)'green eggs and ham'# Tuples>>>('x',)[0]# Indexing a single-item tuple'x' >>>('x', 'y')[1]# Indexing a two-item tuple'y'# Lists>>>L = [1,2,3] + [4,5,6]# List operations>>>L, L[:], L[:0], L[-2], L[-2:]([1, 2, 3, 4, 5, 6], [1, 2, 3, 4, 5, 6], [], 5, [5, 6]) >>>([1,2,3]+[4,5,6])[2:4][3, 4] >>>[L[2], L[3]]# Fetch from offsets; store in a list[3, 4] >>>L.reverse(); L# Method: reverse list in place[6, 5, 4, 3, 2, 1] >>>L.sort(); L# Method: sort list in place[1, 2, 3, 4, 5, 6] >>>L.index(4)# Method: offset of first four (search)3# Dictionaries>>>{'a':1, 'b':2}['b']# Index a dictionary by key2 >>>D = {'x':1, 'y':2, 'z':3}>>>D['w'] = 0# Create a new entry>>>D['x'] + D['w']1 >>>D[(1,2,3)] = 4# A tuple used as a key (immutable)>>>D{'w': 0, 'z': 3, 'y': 2, (1, 2, 3): 4, 'x': 1} >>>list(D.keys()), list(D.values()), (1,2,3) in D# Methods, key test(['w', 'z', 'y', (1, 2, 3), 'x'], [0, 3, 2, 4, 1], True)# Empties>>>[[]], ["",[],(),{},None]# Lots of nothings: empty objects([[]], ['', [], (), {}, None])
L[4]) raises an
error; Python always checks to make sure that all offsets are within
the bounds of a sequence.L[-1000:100]) works because Python scales
out-of-bounds slices so that they always fit (the limits are set to
zero and the sequence length, if required).L[3:1]), doesn’t really work. You get back
an empty slice ([ ]) because Python
scales the slice limits to make sure that the lower bound is always
less than or equal to the upper bound (e.g., L[3:1] is scaled to L[3:3], the empty insertion point at offset
3). Python slices are always
extracted from left to right, even if you use negative indexes (they
are first converted to positive indexes by adding the sequence
length). Note that Python 2.3’s three-limit slices modify this
behavior somewhat. For instance, L[3:1:-1] does extract from right to
left:>>>L = [1, 2, 3, 4]>>>L[4]Traceback (innermost last): File "<stdin>", line 1, in ? IndexError: list index out of range >>>L[-1000:100][1, 2, 3, 4] >>>L[3:1][] >>>L[1, 2, 3, 4] >>>L[3:1] = ['?']>>>L[1, 2, 3, '?', 4]
>>>L = [1,2,3,4]>>>L[2] = []>>>L[1, 2, [], 4] >>>L[2:3] = []>>>L[1, 2, 4] >>>del L[0]>>>L[2, 4] >>>del L[1:]>>>L[2] >>>L[1:2] = 1Traceback (innermost last): File "<stdin>", line 1, in ? TypeError: illegal argument type for built-in operation
X and Y
are swapped. When tuples appear on the left and right of an assignment
symbol (=), Python assigns objects
on the right to targets on the left according to their positions. This
is probably easiest to understand by noting that the targets on the
left aren’t a real tuple, even though they look like one; they are
simply a set of independent assignment targets. The items on the right
are a tuple, which gets unpacked during the assignment (the tuple
provides the temporary assignment needed to achieve the swap
effect):>>>X = 'spam'>>>Y = 'eggs'>>>X, Y = Y, X>>>X'eggs' >>>Y'spam'
>>>D = {}>>>D[1] = 'a'>>>D[2] = 'b'>>>D[(1, 2, 3)] = 'c'>>>D{1: 'a', 2: 'b', (1, 2, 3): 'c'}
D['d']) raises an error;
assigning to a nonexistent key (D['d']='spam') creates a new dictionary
entry. On the other hand, out-of-bounds indexing for lists raises an
error too, but so do out-of-bounds assignments. Variable names work
like dictionary keys; they must have already been assigned when
referenced, but they are created when first assigned. In fact,
variable names can be processed as dictionary keys if you wish
(they’re made visible in module namespace or stack-frame
dictionaries):>>>D = {'a':1, 'b':2, 'c':3}>>>D['a']1 >>>D['d']Traceback (innermost last): File "<stdin>", line 1, in ? KeyError: d >>>D['d'] = 4>>>D{'b': 2, 'd': 4, 'a': 1, 'c': 3} >>> >>>L = [0, 1]>>>L[2]Traceback (innermost last): File "<stdin>", line 1, in ? IndexError: list index out of range >>>L[2] = 3Traceback (innermost last): File "<stdin>", line 1, in ? IndexError: list assignment index out of range
+ operator doesn’t
work on different/mixed types (e.g., string + list, list + tuple).+ doesn’t work for
dictionaries, as they aren’t sequences.append method works
only for lists, not strings, and keys works only on dictionaries.
append assumes its target is
mutable, since it’s an in-place extension; strings are
immutable.>>>"x" + 1Traceback (innermost last): File "<stdin>", line 1, in ? TypeError: illegal argument type for built-in operation >>> >>>{} + {}Traceback (innermost last): File "<stdin>", line 1, in ? TypeError: bad operand type(s) for + >>> >>>[].append(9)>>>"".append('s')Traceback (innermost last): File "<stdin>", line 1, in ? AttributeError: attribute-less object >>> >>>list({}.keys())# list() needed in 3.X, not 2.X[] >>>[].keys()Traceback (innermost last): File "<stdin>", line 1, in ? AttributeError: keys >>> >>>[][:][] >>>""[:]''
S[0][0][0][0][0]
just keeps indexing the first character over and over. This generally
doesn’t work for lists (lists can hold arbitrary objects) unless the
list contains strings:>>>S = "spam">>>S[0][0][0][0][0]'s' >>>L = ['s', 'p']>>>L[0][0][0]'s'
>>>S = "spam">>>S = S[0] + 'l' + S[2:]>>>S'slam' >>>S = S[0] + 'l' + S[2] + S[3]>>>S'slam'
bytearray string type in Chapter 37 — it’s a mutable sequence of small
integers that is essentially processed the same as a string.)>>>me = {'name':('John', 'Q', 'Doe'), 'age':'?', 'job':'engineer'}>>>me['job']'engineer' >>>me['name'][2]'Doe'
ls is a
Unix command; use dir on
Windows):# File: maker.pyfile = open('myfile.txt', 'w') file.write('Hello file world!\n')# Or: open().write()file.close()# close not always needed# File: reader.pyfile = open('myfile.txt')# 'r' is default open modeprint(file.read())# Or print(open().read())%python maker.py%python reader.pyHello file world! %ls -l myfile.txt-rwxrwxrwa 1 0 0 19 Apr 13 16:33 myfile.txt
>>>S = 'spam'>>>for c in S:...print(ord(c))... 115 112 97 109 >>>x = 0>>>for c in S: x += ord(c)# Or: x = x + ord(c)... >>>x433 >>>x = []>>>for c in S: x.append(ord(c))... >>>x[115, 112, 97, 109] >>>list(map(ord, S))# list() required in 3.X, not 2.X[115, 112, 97, 109] >>>[ord(c) for c in S]# map and listcomps automate list builders[115, 112, 97, 109]
\a) 50 times;
assuming your machine can handle it, and when it’s run outside of
IDLE, you may get a series of beeps (or one sustained tone, if your
machine is fast enough). Hey — I warned you.keys and sort calls like this because sort returns None. In Python 2.2 and later, you can
iterate through dictionary keys directly without calling keys (e.g., for key
in D:), but the keys list will not be sorted like it is by
this code. In more recent Pythons, you can achieve the same effect
with the sorted built-in,
too:>>>D = {'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6, 'g':7}>>>D{'f': 6, 'c': 3, 'a': 1, 'g': 7, 'e': 5, 'd': 4, 'b': 2} >>> >>>keys = list(D.keys())# list() required in 3.X, not in 2.X>>>keys.sort()>>>for key in keys:...print(key, '=>', D[key])... a => 1 b => 2 c => 3 d => 4 e => 5 f => 6 g => 7 >>>for key in sorted(D):# Better, in more recent Pythons...print(key, '=>', D[key])
e, assign the result of 2 ** X to a variable outside the loops of
steps a and b, and use it inside the loop. Your results
may vary a bit; this exercise is mostly designed to get you playing
with code alternatives, so anything reasonable gets full
credit:# a
L = [1, 2, 4, 8, 16, 32, 64]
X = 5
i = 0
while i < len(L):
if 2 ** X == L[i]:
print('at index', i)
break
i += 1
else:
print(X, 'not found')
# b
L = [1, 2, 4, 8, 16, 32, 64]
X = 5
for p in L:
if (2 ** X) == p:
print((2 ** X), 'was found at', L.index(p))
break
else:
print(X, 'not found')
# c
L = [1, 2, 4, 8, 16, 32, 64]
X = 5
if (2 ** X) in L:
print((2 ** X), 'was found at', L.index(2 ** X))
else:
print(X, 'not found')
# d
X = 5
L = []
for i in range(7): L.append(2 ** i)
print(L)
if (2 ** X) in L:
print((2 ** X), 'was found at', L.index(2 ** X))
else:
print(X, 'not found')
# f
X = 5
L = list(map(lambda x: 2**x, range(7))) # Or [2**x for x in range(7)]
print(L) # list() to print all in 3.X, not 2.X
if (2 ** X) in L:
print((2 ** X), 'was found at', L.index(2 ** X))
else:
print(X, 'not found')print (and hence your function) is
technically a polymorphic operation, which does
the right thing for each type of object:%python>>>def func(x): print(x)... >>>func("spam")spam >>>func(42)42 >>>func([1, 2, 3])[1, 2, 3] >>>func({'food': 'spam'}){'food': 'spam'}
print
to see results in the test calls because a file isn’t the same as code
typed interactively; Python doesn’t normally echo the results of
expression statements in files:def adder(x, y):
return x + y
print(adder(2, 3))
print(adder('spam', 'eggs'))
print(adder(['a', 'b'], ['c', 'd']))
% python mod.py
5
spameggs
['a', 'b', 'c', 'd']adder functions are shown in the following
file, adders.py. The hard part
here is figuring out how to initialize an accumulator to an empty
value of whatever type is passed in. The first solution uses manual
type testing to look for an integer, and an empty slice of the first
argument (assumed to be a sequence) if the argument is determined not
to be an integer. The second solution uses the first argument to
initialize and scan items 2 and beyond, much like one of the min function variants shown in Chapter 18.+ doesn’t work on mixed types or
dictionaries). You could add a type test and special code to allow
dictionaries, too, but that’s extra credit.def adder1(*args):
print('adder1', end=' ')
if type(args[0]) == type(0): # Integer?
sum = 0 # Init to zero
else: # else sequence:
sum = args[0][:0] # Use empty slice of arg1
for arg in args:
sum = sum + arg
return sum
def adder2(*args):
print('adder2', end=' ')
sum = args[0] # Init to arg1
for next in args[1:]:
sum += next # Add items 2..N
return sum
for func in (adder1, adder2):
print(func(2, 3, 4))
print(func('spam', 'eggs', 'toast'))
print(func(['a', 'b'], ['c', 'd'], ['e', 'f']))
% python adders.py
adder1 9
adder1 spameggstoast
adder1 ['a', 'b', 'c', 'd', 'e', 'f']
adder2 9
adder2 spameggstoast
adder2 ['a', 'b', 'c', 'd', 'e', 'f']**args form in the function
header and use a loop (e.g., for x in
args.keys(): use args[x]), or use args.values() to make this the same as
summing *args positionals:def adder(good=1, bad=2, ugly=3):
return good + bad + ugly
print(adder())
print(adder(5))
print(adder(5, 6))
print(adder(5, 6, 7))
print(adder(ugly=7, good=6, bad=5))
% python mod.py
6
10
14
18
18
# Second part solutions
def adder1(*args): # Sum any number of positional args
tot = args[0]
for arg in args[1:]:
tot += arg
return tot
def adder2(**args): # Sum any number of keyword args
argskeys = list(args.keys()) # list needed in 3.X!
tot = args[argskeys[0]]
for key in argskeys[1:]:
tot += args[key]
return tot
def adder3(**args): # Same, but convert to list of values
args = list(args.values()) # list needed to index in 3.X!
tot = args[0]
for arg in args[1:]:
tot += arg
return tot
def adder4(**args): # Same, but reuse positional version
return adder1(*args.values())
print(adder1(1, 2, 3), adder1('aa', 'bb', 'cc'))
print(adder2(a=1, b=2, c=3), adder2(a='aa', b='bb', c='cc'))
print(adder3(a=1, b=2, c=3), adder3(a='aa', b='bb', c='cc'))
print(adder4(a=1, b=2, c=3), adder4(a='aa', b='bb', c='cc'))D.copy() and D1.update(D2) to handle things like copying
and adding (merging) dictionaries. See Chapter 8 for dict.update examples, and Python’s library
manual or O’Reilly’s Python Pocket
Reference for more details. X[:] doesn’t work for dictionaries, as
they’re not sequences (see Chapter 8
for details). Also, remember that if you assign (e = d) rather than copying, you generate a
reference to a shared dictionary object; changing
d changes e, too:def copyDict(old):
new = {}
for key in old.keys():
new[key] = old[key]
return new
def addDict(d1, d2):
new = {}
for key in d1.keys():
new[key] = d1[key]
for key in d2.keys():
new[key] = d2[key]
return new
% python
>>> from dicts import *
>>> d = {1: 1, 2: 2}
>>> e = copyDict(d)
>>> d[2] = '?'
>>> d
{1: 1, 2: '?'}
>>> e
{1: 1, 2: 2}
>>> x = {1: 1}
>>> y = {2: 2}
>>> z = addDict(x, y)
>>> z
{1: 1, 2: 2}def f1(a, b): print(a, b)# Normal argsdef f2(a, *b): print(a, b)# Positional varargsdef f3(a, **b): print(a, b)# Keyword varargsdef f4(a, *b, **c): print(a, b, c)# Mixed modesdef f5(a, b=2, c=3): print(a, b, c)# Defaultsdef f6(a, b=2, *c): print(a, b, c)# Defaults and positional varargs%python>>>f1(1, 2)# Matched by position (order matters)1 2 >>>f1(b=2, a=1)# Matched by name (order doesn't matter)1 2 >>>f2(1, 2, 3)# Extra positionals collected in a tuple1 (2, 3) >>>f3(1, x=2, y=3)# Extra keywords collected in a dictionary1 {'x': 2, 'y': 3} >>>f4(1, 2, 3, x=2, y=3)# Extra of both kinds1 (2, 3) {'x': 2, 'y': 3} >>>f5(1)# Both defaults kick in1 2 3 >>>f5(1, 4)# Only one default used1 4 3 >>>f6(1)# One argument: matches "a"1 2 () >>>f6(1, 3, 4)# Extra positional collected1 3 (4,)
if test to trap
negatives, 0, and 1. I also changed / to //
in this edition to make this solution immune to the Python 3.X
/ true division changes we studied
in Chapter 5, and to enable it to support
floating-point numbers (uncomment the from statement and change // to /
to see the differences in 2.X):#from __future__ import division
def prime(y):
if y <= 1: # For some y > 1
print(y, 'not prime')
else:
x = y // 2 # 3.X / fails
while x > 1:
if y % x == 0: # No remainder?
print(y, 'has factor', x)
break # Skip else
x -= 1
else:
print(y, 'is prime')
prime(13); prime(13.0)
prime(15); prime(15.0)
prime(3); prime(2)
prime(1); prime(-3)// operator allows it to work for
floating-point numbers too, even though it perhaps should not:% python primes.py
13 is prime
13.0 is prime
15 has factor 5
15.0 has factor 5.0
3 is prime
2 is prime
1 not prime
-3 not primefor loop over range(y, 1, −1) may be a bit quicker than
the while, but the algorithm is the
real bottleneck here.) To time alternatives, use the homegrown
timer or standard library timeit modules and coding patterns like
those used in Chapter 21’s timing
sections (and see Solution 10).>>>values = [2, 4, 9, 16, 25]>>>import math>>>res = []>>>for x in values: res.append(math.sqrt(x))... >>>res[1.4142135623730951, 2.0, 3.0, 4.0, 5.0] >>>list(map(math.sqrt, values))[1.4142135623730951, 2.0, 3.0, 4.0, 5.0] >>>[math.sqrt(x) for x in values][1.4142135623730951, 2.0, 3.0, 4.0, 5.0] >>>list(math.sqrt(x) for x in values)[1.4142135623730951, 2.0, 3.0, 4.0, 5.0]
# File timer2.py (2.X and 3.X)...same as listed in Chapter 21...# File timesqrt.py import sys, timer2 reps = 10000 repslist = range(reps)# Pull out range list time for 2.Xfrom math import sqrt# Not math.sqrt: adds attr fetch timedef mathMod(): for i in repslist: res = sqrt(i) return res def powCall(): for i in repslist: res = pow(i, .5) return res def powExpr(): for i in repslist: res = i ** .5 return res print(sys.version) for test in (mathMod, powCall, powExpr): elapsed, result = timer2.bestoftotal(test, _reps1=3, _reps=1000) print ('%s: %.5f => %s' % (test.__name__, elapsed, result))
math module is
quicker than the ** expression,
which is quicker than the pow call;
however, you should try this with your code and on your own machine
and version of Python. Also, note that Python 3.3 is essentially twice
as slow as 2.7 on this test, and PyPy is a rough order of magnitude
(10X) faster than both CPythons, despite the fact that this is running
floating-point math and iterations. Later versions of any of these
Pythons might differ, so time this in the future to see for
yourself:c:\code>py −3 timesqrt.py3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] mathMod: 2.04481 => 99.99499987499375 powCall: 3.40973 => 99.99499987499375 powExpr: 2.56458 => 99.99499987499375 c:\code>py −2 timesqrt.py2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] mathMod: 1.04337 => 99.994999875 powCall: 2.57516 => 99.994999875 powExpr: 1.89560 => 99.994999875 c:\code>c:\pypy\pypy-1.9\pypy timesqrt.py2.7.2 (341e1e3821ff, Jun 07 2012, 15:43:00) [PyPy 1.9.0 with MSC v.1500 32 bit] mathMod: 0.07491 => 99.994999875 powCall: 0.85678 => 99.994999875 powExpr: 0.85453 => 99.994999875
for loops interactively, you can run a
session like the following. It appears that the two are roughly the
same in this regard under Python 3.3; unlike list comprehensions,
though, manual loops are slightly faster than dictionary
comprehensions today (though the difference isn’t exactly
earth-shattering — at the end we save half a second when making 50
dictionaries of 1,000,000 items each). Again, rather than taking these
results as gospel you should investigate further on your own, on your
computer and with your Python:C:\code>c:\python33\python>>> >>>def dictcomp(I):return {i: i for i in range(I)}>>>def dictloop(I):new = {}for i in range(I): new[i] = ireturn new>>>dictcomp(10){0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} >>>dictloop(10){0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} >>> >>>from timer2 import total, bestof>>>bestof(dictcomp, 10000)[0]# 10,000-item dict0.0017095345403959072 >>>bestof(dictloop, 10000)[0]0.002097576400046819 >>> >>>bestof(dictcomp, 100000)[0]# 100,000-items: 10X slower0.012716923463358398 >>>bestof(dictloop, 100000)[0]0.014129806355413166 >>> >>>bestof(dictcomp, 1000000)[0]# 1 of 1M-items: 10X time0.11614425187337929 >>>bestof(dictloop, 1000000)[0]0.1331144855439561 >>> >>>total(dictcomp, 1000000, _reps=50)[0]# Total to make 50 1M-item dicts5.8162020671780965 >>>total(dictloop, 1000000, _reps=50)[0]6.626680761285343
range,
comprehension, or map will do the
job here as well, but recursion is useful enough to experiment with
here (print is a function in 3.X
only, unless you import it from __future__ or code your own
equivalent):def countdown(N):
if N == 0:
print('stop') # 2.X: print 'stop'
else:
print(N, end=' ') # 2.X: print N,
countdown(N-1)
>>> countdown(5)
5 4 3 2 1 stop
>>> countdown(20)
20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 stop
# Nonrecursive options:
>>> list(range(5, 0, −1))
[5, 4, 3, 2, 1]
# On 3.X only:
>>> t = [print(i, end=' ') for i in range(5, 0, −1)]
5 4 3 2 1
>>> t = list(map(lambda x: print(x, end=' '), range(5, 0, −1)))
5 4 3 2 1for or yield from here (yield from works in 3.3 and later
only):def countdown2(N):# Generator function, recursiveif N == 0: yield 'stop' else: yield N for x in countdown2(N-1): yield x# 3.3+: yield from countdown2(N-1)>>>list(countdown2(5))[5, 4, 3, 2, 1, 'stop']# Nonrecursive options:>>>def countdown3():# Generator function, simpleryield from range(5, 0, −1)# Pre 3.3: for x in range(): yield x>>>list(countdown3())[5, 4, 3, 2, 1] >>>list(x for x in range(5, 0, −1))# Equivalent generator expression[5, 4, 3, 2, 1] >>>list(range(5, 0, −1))# Equivalent nongenerator form[5, 4, 3, 2, 1]
2..N+1 to skip
an iteration, and fact2 could use reduce(operator.mul, range(N, 1, −1)) to avoid a lambda.#!python from __future__ import print_function# File factorials.pyfrom functools import reduce from timeit import repeat import math def fact0(N):# Recursiveif N == 1:# Fails at 999 by defaultreturn N else: return N * fact0(N-1) def fact1(N): return N if N == 1 else N * fact1(N-1)# Recursive, one-linerdef fact2(N):# Functionalreturn reduce(lambda x, y: x * y, range(1, N+1)) def fact3(N): res = 1 for i in range(1, N+1): res *= i# Iterativereturn res def fact4(N): return math.factorial(N)# Stdlib "batteries"# Tests print(fact0(6), fact1(6), fact2(6), fact3(6), fact4(6))# 6*5*4*3*2*1: all 720print(fact0(500) == fact1(500) == fact2(500) == fact3(500) == fact4(500))# Truefor test in (fact0, fact1, fact2, fact3, fact4): print(test.__name__, min(repeat(stmt=lambda: test(500), number=20, repeat=3))) r""" C:\code> py −3 factorials.py 720 720 720 720 720 True fact0 0.003990868798355564 fact1 0.003901433457907475 fact2 0.002732909419593966 fact3 0.002052614370939676 fact4 0.0003401475243271501 """
N reaches 999 due to the
default stack size setting in sys;
per Chapter 19, this limit can be
increased, but simple loops or the standard library tool seem the best
route here in any event.''.join(reversed(S)) may be the preferred
way to reverse a string, even though recursive solutions are possible.
Time the following to see how: as for factorials in 3.X, recursion is
today an order of magnitude slower in CPython, though these results vary in PyPy:def rev1(S):
if len(S) == 1:
return S
else:
return S[-1] + rev1(S[:-1]) # Recursive: 10x slower in CPython today
def rev2(S):
return ''.join(reversed(S)) # Nonrecursive iterable: simpler, faster
def rev3(S):
return S[::-1] # Even better?: sequence reversal by slice
len built-in returns the lengths of strings
and lists:def countLines(name):
file = open(name)
return len(file.readlines())
def countChars(name):
return len(open(name).read())
def test(name): # Or pass file object
return countLines(name), countChars(name) # Or return a dictionary
% python
>>> import mymod
>>> mymod.test('mymod.py')
(10, 291)def countLines(name):
tot = 0
for line in open(name): tot += 1
return tot
def countChars(name):
tot = 0
for line in open(name): tot += len(line)
return totdef countlines(name): return sum(+1 for line in open(name)) def countchars(name): return sum(len(line) for line in open(name))
wc command; on Windows, right-click on your
file to view its properties. Note that your script may report fewer
characters than Windows does — for portability, Python converts Windows
\r\n line-end markers to \n, thereby dropping 1 byte (character) per
line. To match byte counts with Windows exactly, you must open in
binary mode ('rb'), or add the
number of bytes corresponding to the number of lines. See Chapter 9 and Chapter 37 for more on end-of-line
translations in text files.seek method of the built-in file object. It
works like C’s fseek call (and may
call it behind the scenes): seek
resets the current position in the file to a passed-in offset. After a
seek, future input/output
operations are relative to the new position. To rewind to the start of
a file without closing and reopening it, call file.seek(0); the file read methods all pick up at the current
position in the file, so you need to rewind to reread. Here’s what
this tweak would look like:def countLines(file):
file.seek(0) # Rewind to start of file
return len(file.readlines())
def countChars(file):
file.seek(0) # Ditto (rewind if needed)
return len(file.read())
def test(name):
file = open(name) # Pass file object
return countLines(file), countChars(file) # Open file only once
>>> import mymod2
>>> mymod2.test("mymod2.py")
(11, 392)from/from *. Here’s the from * part; replace * with countChars to do the rest:%python>>>from mymod import *>>>countChars("mymod.py")291
__main__. If you code it
properly, this file works in either mode — program run or module
import:def countLines(name):
file = open(name)
return len(file.readlines())
def countChars(name):
return len(open(name).read())
def test(name): # Or pass file object
return countLines(name), countChars(name) # Or return a dictionary
if __name__ == '__main__':
print(test('mymod.py'))
% python mymod.py
(13, 346)sys.argv, and Chapter 10 for more on input — and use raw_input instead in 2.X):if __name__ == '__main__':
print(test(input('Enter file name:')) # Console (raw_input in 2.X)
if __name__ == '__main__':
import sys # Command line
print(test(sys.argv[1]))from mymod import countLines, countChars
print(countLines('mymod.py'), countChars('mymod.py'))
% python myclient.py
13 346mymod’s functions are accessible (that is,
importable) from the top level of myclient, since from simply assigns to names in the importer
(it works as if mymod’s defs appeared in myclient). For example, another file can
say:import myclient myclient.countLines(...) from myclient import countChars countChars(...)
myclient used import instead of from, you’d need to use a path to get to the
functions in mymod through myclient:import myclient myclient.mymod.countLines(...) from myclient import mymod mymod.countChars(...)
somename — mod1.somename, collector.somename, and __main__.somename; all three share the same
integer object initially, and only the name somename exists at the interactive prompt as
is:# File mod1.pysomename = 42# File collector.pyfrom mod1 import *# Collect lots of names herefrom mod2 import *# from assigns to my namesfrom mod3 import * >>>from collector import somename
cp and vi instead of copy and notepad). This works in any directory (I’m
using my own code directory here), and you can do some of this from a
file explorer GUI, too.print statement coded in the directory’s
initialization file fires only the first time it is imported, not the second;
raw strings are also used here to avoid escape issues in the file paths:C:\code>mkdir mypkgC:\code>copy mymod.py mypkg\mymod.pyC:\code>notepad mypkg\__init__.py...coded a print statement...C:\code>python>>>import mypkg.mymodinitializing mypkg >>>mypkg.mymod.countLines(r'mypkg\mymod.py')13 >>>from mypkg.mymod import countChars>>>countChars(r'mypkg\mymod.py')346
recur2 first works
because the recursive import then happens at the import in recur1, not at a from in recur2.recur2 first works because the recursive
import from recur1 to recur2 fetches recur2 as a whole, instead of getting
specific names. recur2 is
incomplete when it’s imported from recur1, but because it uses import instead of from, you’re safe: Python finds and returns
the already created recur2 module
object and continues to run the rest of recur1 without a glitch. When the recur2 import resumes, the second from finds the name Y in recur1 (it’s been run completely), so no
error is reported.import or from in the script interactively. For
instance, running recur1 as a
script works, because it is the same as importing recur2 interactively, as recur2 is the first module imported in
recur1. Running recur2 as a script fails for the same
reason — it’s the same as running its first import interactively.__add__ overload has to
appear only once, in the superclass, as it invokes type-specific
add methods in subclasses:class Adder:
def add(self, x, y):
print('not implemented!')
def __init__(self, start=[]):
self.data = start
def __add__(self, other): # Or in subclasses?
return self.add(self.data, other) # Or return type?
class ListAdder(Adder):
def add(self, x, y):
return x + y
class DictAdder(Adder):
def add(self, x, y):
new = {}
for k in x.keys(): new[k] = x[k]
for k in y.keys(): new[k] = y[k]
return new
% python
>>> from adder import *
>>> x = Adder()
>>> x.add(1, 2)
not implemented!
>>> x = ListAdder()
>>> x.add([1], [2])
[1, 2]
>>> x = DictAdder()
>>> x.add({1:1}, {2:2})
{1: 1, 2: 2}
>>> x = Adder([1])
>>> x + [2]
not implemented!
>>>
>>> x = ListAdder([1])
>>> x + [2]
[1, 2]
>>> [2] + x
In 3.3: TypeError: can only concatenate list (not "ListAdder") to list
Earlier: TypeError: __add__ nor __radd__ defined for these operands+; if you want to fix this, use __radd__ methods, as described in “Operator
Overloading” in Chapter 30.add method to take
just one argument, in the spirit of other examples in this part of the
book (this is adder2.py):class Adder:
def __init__(self, start=[]):
self.data = start
def __add__(self, other): # Pass a single argument
return self.add(other) # The left side is in self
def add(self, y):
print('not implemented!')
class ListAdder(Adder):
def add(self, y):
return self.data + y
class DictAdder(Adder):
def add(self, y):
d = self.data.copy() # Change to use self.data instead of x
d.update(y) # Or "cheat" by using quicker built-ins
return d
x = ListAdder([1, 2, 3])
y = x + [4, 5, 6]
print(y) # Prints [1, 2, 3, 4, 5, 6]
z = DictAdder(dict(name='Bob')) + {'a':1}
print(z) # Prints {'name': 'Bob', 'a': 1}add altogether and simply define
type-specific __add__ methods in
the two subclasses.__getattr__ method routes calls to the
wrapped list. For hints on an easier way to code this in Python 2.2
and later, see “Extending Types by Subclassing” in
Chapter 32:class MyList:
def __init__(self, start):
#self.wrapped = start[:] # Copy start: no side effects
self.wrapped = list(start) # Make sure it's a list here
def __add__(self, other):
return MyList(self.wrapped + other)
def __mul__(self, time):
return MyList(self.wrapped * time)
def __getitem__(self, offset): # Also passed a slice in 3.X
return self.wrapped[offset] # For iteration if no __iter__
def __len__(self):
return len(self.wrapped)
def __getslice__(self, low, high): # Ignored in 3.X: uses __getitem__
return MyList(self.wrapped[low:high])
def append(self, node):
self.wrapped.append(node)
def __getattr__(self, name): # Other methods: sort/reverse/etc
return getattr(self.wrapped, name)
def __repr__(self): # Catchall display method
return repr(self.wrapped)
if __name__ == '__main__':
x = MyList('spam')
print(x)
print(x[2])
print(x[1:])
print(x + ['eggs'])
print(x * 3)
x.append('a')
x.sort()
print(' '.join(c for c in x))
c:\code> python mylist.py
['s', 'p', 'a', 'm']
a
['p', 'a', 'm']
['s', 'p', 'a', 'm', 'eggs']
['s', 'p', 'a', 'm', 's', 'p', 'a', 'm', 's', 'p', 'a', 'm']
a a m p slist instead of slicing here,
because otherwise the result may not be a true list and so will not
respond to expected list methods, such as append (e.g., slicing a string returns
another string, not a list). You would be able to copy a MyList start value by slicing because its
class overloads the slicing operation and provides the expected list
interface; however, you need to avoid slice-based copying for objects
such as strings.from mylist import MyList
class MyListSub(MyList):
calls = 0 # Shared by instances
def __init__(self, start):
self.adds = 0 # Varies in each instance
MyList.__init__(self, start)
def __add__(self, other):
print('add: ' + str(other))
MyListSub.calls += 1 # Class-wide counter
self.adds += 1 # Per-instance counts
return MyList.__add__(self, other)
def stats(self):
return self.calls, self.adds # All adds, my adds
if __name__ == '__main__':
x = MyListSub('spam')
y = MyListSub('foo')
print(x[2])
print(x[1:])
print(x + ['eggs'])
print(x + ['toast'])
print(y + ['bar'])
print(x.stats())
c:\code> python mysub.py
a
['p', 'a', 'm']
add: ['eggs']
['s', 'p', 'a', 'm', 'eggs']
add: ['toast']
['s', 'p', 'a', 'm', 'toast']
add: ['bar']
['f', 'o', 'o', 'bar']
(3, 2)__getattr__, too; you need to return a value
to make them work. As noted in Chapter 32 and elsewhere, __getattr__ is not
called for built-in operations in Python 3.X (and in 2.X if new-style
classes are used), so the expressions aren’t intercepted at all here;
in new-style classes, a class like this must redefine __X__ operator overloading methods explicitly.
More on this in Chapter 28, Chapter 31, Chapter 32, Chapter 38, and Chapter 39: it
can impact much code!c:\code>py −2>>>class Attrs:def __getattr__(self, name):print('get %s' % name)def __setattr__(self, name, value):print('set %s %s' % (name, value))>>>x = Attrs()>>>x.appendget append >>>x.spam = 'pork'set spam pork >>>x + 2get __coerce__ TypeError: 'NoneType' object is not callable >>>x[1]get __getitem__ TypeError: 'NoneType' object is not callable >>>x[1:5]get __getslice__ TypeError: 'NoneType' object is not callable c:\code>py −3>>>...same startup code...>>>x + 2TypeError: unsupported operand type(s) for +: 'Attrs' and 'int' >>>x[1]TypeError: 'Attrs' object does not support indexing >>>x[1:5]TypeError: 'Attrs' object is not subscriptable
%python>>>from setwrapper import Set>>>x = Set([1, 2, 3, 4])# Runs __init__>>>y = Set([3, 4, 5])>>>x & y# __and__, intersect, then __repr__Set:[3, 4] >>>x | y# __or__, union, then __repr__Set:[1, 2, 3, 4, 5] >>>z = Set("hello")# __init__ removes duplicates>>>z[0], z[-1], z[2:]# __getitem__('h', 'o', ['l', 'o']) >>>for c in z: print(c, end=' ')# __iter__ (else __getitem__) [3.X print]... h e l o >>>''.join(c.upper() for c in z)# __iter__ (else __getitem__)'HELO' >>>len(z), z# __len__, __repr__(4, Set:['h', 'e', 'l', 'o']) >>>z & "mello", z | "mello"(Set:['e', 'l', 'o'], Set:['h', 'e', 'l', 'o', 'm'])
from setwrapper import Set
class MultiSet(Set):
"""
Inherits all Set names, but extends intersect and union to support
multiple operands; note that "self" is still the first argument
(stored in the *args argument now); also note that the inherited
& and | operators call the new methods here with 2 arguments, but
processing more than 2 requires a method call, not an expression;
intersect doesn't remove duplicates here: the Set constructor does;
"""
def intersect(self, *others):
res = []
for x in self: # Scan first sequence
for other in others: # For all other args
if x not in other: break # Item in each one?
else: # No: break out of loop
res.append(x) # Yes: add item to end
return Set(res)
def union(*args): # self is args[0]
res = []
for seq in args: # For all args
for x in seq: # For all nodes
if not x in res:
res.append(x) # Add new items to result
return Set(res)& or calling intersect, but you must call intersect for three or more operands;
& is a binary (two-sided)
operator. Also, note that we could have called MultiSet simply Set to make this change more transparent if
we used setwrapper.Set to refer to
the original within multiset (the
as clause in an import could rename
the class too if desired):>>>from multiset import *>>>x = MultiSet([1, 2, 3, 4])>>>y = MultiSet([3, 4, 5])>>>z = MultiSet([0, 1, 2])>>>x & y, x | y# Two operands(Set:[3, 4], Set:[1, 2, 3, 4, 5]) >>>x.intersect(y, z)# Three operandsSet:[] >>>x.union(y, z)Set:[1, 2, 3, 4, 5, 0] >>>x.intersect([1,2,3], [2,3,4], [1,2,3])# Four operandsSet:[2, 3] >>>x.union(range(10))# Non-MultiSets work, tooSet:[1, 2, 3, 4, 0, 5, 6, 7, 8, 9] >>>w = MultiSet('spam')# String sets>>>wSet:['s', 'p', 'a', 'm'] >>>''.join(w | 'super')'spamuer' >>>(w | 'super') & MultiSet('slots')Set:['s']
dir-based version, and
also do this when formatting class objects in the tree climber
variant:class ListInstance:
def __attrnames(self):
...unchanged...
def __str__(self):
return '<Instance of %s(%s), address %s:\n%s>' % (
self.__class__.__name__, # My class's name
self.__supers(), # My class's own supers
id(self), # My address
self.__attrnames()) # name=value list
def __supers(self):
names = []
for super in self.__class__.__bases__: # One level up from class
names.append(super.__name__) # name, not str(super)
return ', '.join(names)
# Or: ', '.join(super.__name__ for super in self.__class__.__bases__)
c:\code> py listinstance-exercise.py
<Instance of Sub(Super, ListInstance), address 43671000:
data1=spam
data2=eggs
data3=42
>class Lunch:
def __init__(self): # Make/embed Customer, Employee
self.cust = Customer()
self.empl = Employee()
def order(self, foodName): # Start Customer order simulation
self.cust.placeOrder(foodName, self.empl)
def result(self): # Ask the Customer about its Food
self.cust.printFood()
class Customer:
def __init__(self): # Initialize my food to None
self.food = None
def placeOrder(self, foodName, employee): # Place order with Employee
self.food = employee.takeOrder(foodName)
def printFood(self): # Print the name of my food
print(self.food.name)
class Employee:
def takeOrder(self, foodName): # Return Food, with desired name
return Food(foodName)
class Food:
def __init__(self, name): # Store food name
self.name = name
if __name__ == '__main__':
x = Lunch() # Self-test code
x.order('burritos') # If run, not imported
x.result()
x.order('pizza')
x.result()
% python lunch.py
burritos
pizzaself.speak reference in Animal triggers an independent inheritance
search, which finds speak in a
subclass. Test this interactively per the exercise description. Try
extending this hierarchy with new classes, and making instances of
various classes in the tree:class Animal:
def reply(self): self.speak() # Back to subclass
def speak(self): print('spam') # Custom message
class Mammal(Animal):
def speak(self): print('huh?')
class Cat(Mammal):
def speak(self): print('meow')
class Dog(Mammal):
def speak(self): print('bark')
class Primate(Mammal):
def speak(self): print('Hello world!')
class Hacker(Primate): pass # Inherit from Primateline method in the Actor superclass works: by accessing
self attributes twice, it sends
Python back to the instance twice, and hence invokes
two inheritance searches — self.name and self.says() find information in the specific
subclasses:class Actor:
def line(self): print(self.name + ':', repr(self.says()))
class Customer(Actor):
name = 'customer'
def says(self): return "that's one ex-bird!"
class Clerk(Actor):
name = 'clerk'
def says(self): return "no it isn't..."
class Parrot(Actor):
name = 'parrot'
def says(self): return None
class Scene:
def __init__(self):
self.clerk = Clerk() # Embed some instances
self.customer = Customer() # Scene is a composite
self.subject = Parrot()
def action(self):
self.customer.line() # Delegate to embedded
self.clerk.line()
self.subject.line()try/except. My version of the oops function
(file oops.py) follows. As for
the noncoding questions, changing oops to raise a KeyError instead of an IndexError means that the try handler won’t catch the exception — it
“percolates” to the top level and triggers Python’s default error
message. The names KeyError and
IndexError come from the outermost
built-in names scope (the B in “LEGB”). Import
builtins in 3.X (and __builtin__ in Python 2.X) and pass it as an
argument to the dir function to see
this for yourself.def oops():
raise IndexError()
def doomed():
try:
oops()
except IndexError:
print('caught an index error!')
else:
print('no error caught...')
if __name__ == '__main__': doomed()
% python oops.py
caught an index error!from __future__ import print_function # 2.X
class MyError(Exception): pass
def oops():
raise MyError('Spam!')
def doomed():
try:
oops()
except IndexError:
print('caught an index error!')
except MyError as data:
print('caught error:', MyError, data)
else:
print('no error caught...')
if __name__ == '__main__':
doomed()
% python oops2.py
caught error: <class '__main__.MyError'> Spam!as variable data; the error message shows both the class
(<...>) and its instance
(Spam!). The instance must be
inheriting both an __init__ and a
__repr__ or __str__ from Python’s Exception class, or it would print much like
the class does. See Chapter 35 for details
on how this works in built-in exception classes.except and sys.exc_info approach used here will catch
exit-related exceptions that listing Exception with an as variable won’t; that’s probably not ideal
in most applications code, but might be useful in a tool like this
designed to work as a sort of exceptions firewall.import sys, traceback
def safe(callee, *pargs, **kargs):
try:
callee(*pargs, **kargs) # Catch everything else
except: # Or "except Exception as E:"
traceback.print_exc()
print('Got %s %s' % (sys.exc_info()[0], sys.exc_info()[1]))
if __name__ == '__main__':
import oops2
safe(oops2.oops)
c:\code> py −3 exctools.py
Traceback (most recent call last):
File "C:\code\exctools.py", line 5, in safe
callee(*pargs, **kargs) # Catch everything else
File "C:\code\oops2.py", line 6, in oops
raise MyError('Spam!')
oops2.MyError: Spam!
Got <class 'oops2.MyError'> Spam!import sys, traceback
def safe(callee):
def callproxy(*pargs, **kargs):
try:
return callee(*pargs, **kargs)
except:
traceback.print_exc()
print('Got %s %s' % (sys.exc_info()[0], sys.exc_info()[1]))
raise
return callproxy
if __name__ == '__main__':
import oops2
@safe
def test():
oops2.oops()
test()# Find the largest Python source file in a single directoryimport os, glob dirname = r'C:\Python33\Lib' allsizes = [] allpy = glob.glob(dirname + os.sep + '*.py') for filename in allpy: filesize = os.path.getsize(filename) allsizes.append((filesize, filename)) allsizes.sort() print(allsizes[:2]) print(allsizes[-2:])# Find the largest Python source file in an entire directory treeimport sys, os, pprint if sys.platform[:3] == 'win': dirname = r'C:\Python33\Lib' else: dirname = '/usr/lib/python' allsizes = [] for (thisDir, subsHere, filesHere) in os.walk(dirname): for filename in filesHere: if filename.endswith('.py'): fullname = os.path.join(thisDir, filename) fullsize = os.path.getsize(fullname) allsizes.append((fullsize, fullname)) allsizes.sort() pprint.pprint(allsizes[:2]) pprint.pprint(allsizes[-2:])# Find the largest Python source file on the module import search pathimport sys, os, pprint visited = {} allsizes = [] for srcdir in sys.path: for (thisDir, subsHere, filesHere) in os.walk(srcdir): thisDir = os.path.normpath(thisDir) if thisDir.upper() in visited: continue else: visited[thisDir.upper()] = True for filename in filesHere: if filename.endswith('.py'): pypath = os.path.join(thisDir, filename) try: pysize = os.path.getsize(pypath) except: print('skipping', pypath) allsizes.append((pysize, pypath)) allsizes.sort() pprint.pprint(allsizes[:3]) pprint.pprint(allsizes[-3:])# Sum columns in a text file separated by commasfilename = 'data.txt' sums = {} for line in open(filename): cols = line.split(',') nums = [int(col) for col in cols] for (ix, num) in enumerate(nums): sums[ix] = sums.get(ix, 0) + num for key in sorted(sums): print(key, '=', sums[key])# Similar to prior, but using lists instead of dictionaries for sumsimport sys filename = sys.argv[1] numcols = int(sys.argv[2]) totals = [0] * numcols for line in open(filename): cols = line.split(',') nums = [int(x) for x in cols] totals = [(x + y) for (x, y) in zip(totals, nums)] print(totals)# Test for regressions in the output of a set of scriptsimport os testscripts = [dict(script='test1.py', args=''),# Or glob script/args dirdict(script='test2.py', args='spam')] for testcase in testscripts: commandline = '%(script)s %(args)s' % testcase output = os.popen(commandline).read() result = testcase['script'] + '.result' if not os.path.exists(result): open(result, 'w').write(output) print('Created:', result) else: priorresult = open(result).read() if output != priorresult: print('FAILED:', testcase['script']) print(output) else: print('Passed:', testcase['script'])# Build GUI with tkinter (Tkinter in 2.X) with buttons that change color and growfrom tkinter import *# Use Tkinter in 2.Ximport random fontsize = 25 colors = ['red', 'green', 'blue', 'yellow', 'orange', 'white', 'cyan', 'purple'] def reply(text): print(text) popup = Toplevel() color = random.choice(colors) Label(popup, text='Popup', bg='black', fg=color).pack() L.config(fg=color) def timer(): L.config(fg=random.choice(colors)) win.after(250, timer) def grow(): global fontsize fontsize += 5 L.config(font=('arial', fontsize, 'italic')) win.after(100, grow) win = Tk() L = Label(win, text='Spam', font=('arial', fontsize, 'italic'), fg='yellow', bg='navy', relief=RAISED) L.pack(side=TOP, expand=YES, fill=BOTH) Button(win, text='press', command=(lambda: reply('red'))).pack(side=BOTTOM, fill=X) Button(win, text='timer', command=timer).pack(side=BOTTOM, fill=X) Button(win, text='grow', command=grow).pack(side=BOTTOM, fill=X) win.mainloop()# Similar to prior, but use classes so each window has own state informationfrom tkinter import * import random class MyGui: """ A GUI with buttons that change color and make the label grow """ colors = ['blue', 'green', 'orange', 'red', 'brown', 'yellow'] def __init__(self, parent, title='popup'): parent.title(title) self.growing = False self.fontsize = 10 self.lab = Label(parent, text='Gui1', fg='white', bg='navy') self.lab.pack(expand=YES, fill=BOTH) Button(parent, text='Spam', command=self.reply).pack(side=LEFT) Button(parent, text='Grow', command=self.grow).pack(side=LEFT) Button(parent, text='Stop', command=self.stop).pack(side=LEFT) def reply(self): "change the button's color at random on Spam presses" self.fontsize += 5 color = random.choice(self.colors) self.lab.config(bg=color, font=('courier', self.fontsize, 'bold italic')) def grow(self): "start making the label grow on Grow presses" self.growing = True self.grower() def grower(self): if self.growing: self.fontsize += 5 self.lab.config(font=('courier', self.fontsize, 'bold')) self.lab.after(500, self.grower) def stop(self): "stop the button growing on Stop presses" self.growing = False class MySubGui(MyGui): colors = ['black', 'purple']# Customize to change color choicesMyGui(Tk(), 'main') MyGui(Toplevel()) MySubGui(Toplevel()) mainloop()# Email inbox scanning and maintenance utility""" scan pop email box, fetching just headers, allowing deletions without downloading the complete message """ import poplib, getpass, sys mailserver = 'your pop email server name here'# pop.server.netmailuser = 'your pop email user name here' mailpasswd = getpass.getpass('Password for %s?' % mailserver) print('Connecting...') server = poplib.POP3(mailserver) server.user(mailuser) server.pass_(mailpasswd) try: print(server.getwelcome()) msgCount, mboxSize = server.stat() print('There are', msgCount, 'mail messages, size ', mboxSize) msginfo = server.list() print(msginfo) for i in range(msgCount): msgnum = i+1 msgsize = msginfo[1][i].split()[1] resp, hdrlines, octets = server.top(msgnum, 0)# Get hdrs onlyprint('-'*80) print('[%d: octets=%d, size=%s]' % (msgnum, octets, msgsize)) for line in hdrlines: print(line) if input('Print?') in ['y', 'Y']: for line in server.retr(msgnum)[1]: print(line)# Get whole msgif input('Delete?') in ['y', 'Y']: print('deleting') server.dele(msgnum)# Delete on srvrelse: print('skipping') finally: server.quit()# Make sure we unlock mboxinput('Bye.')# Keep window up on Windows# CGI server-side script to interact with a web browser#!/usr/bin/python import cgi form = cgi.FieldStorage()# Parse form dataprint("Content-type: text/html\n")# hdr plus blank lineprint("<HTML>") print("<title>Reply Page</title>")# HTML reply pageprint("<BODY>") if not 'user' in form: print("<h1>Who are you?</h1>") else: print("<h1>Hello <i>%s</i>!</h1>" % cgi.escape(form['user'].value)) print("</BODY></HTML>")# Database script to populate a shelve with Python objects# see also Chapter 28 shelve and Chapter 31 pickle examplesrec1 = {'name': {'first': 'Bob', 'last': 'Smith'}, 'job': ['dev', 'mgr'], 'age': 40.5} rec2 = {'name': {'first': 'Sue', 'last': 'Jones'}, 'job': ['mgr'], 'age': 35.0} import shelve db = shelve.open('dbfile') db['bob'] = rec1 db['sue'] = rec2 db.close()# Database script to print and update shelve created in prior scriptimport shelve db = shelve.open('dbfile') for key in db: print(key, '=>', db[key]) bob = db['bob'] bob['age'] += 1 db['bob'] = bob db.close()# Database script to populate and query a MySql databasefrom MySQLdb import Connect conn = Connect(host='localhost', user='root', passwd='XXXXXXX') curs = conn.cursor() try: curs.execute('drop database testpeopledb') except: pass# Did not existcurs.execute('create database testpeopledb') curs.execute('use testpeopledb') curs.execute('create table people (name char(30), job char(10), pay int(4))') curs.execute('insert people values (%s, %s, %s)', ('Bob', 'dev', 50000)) curs.execute('insert people values (%s, %s, %s)', ('Sue', 'dev', 60000)) curs.execute('insert people values (%s, %s, %s)', ('Ann', 'mgr', 40000)) curs.execute('select * from people') for row in curs.fetchall(): print(row) curs.execute('select * from people where name = %s', ('Bob',)) print(curs.description) colnames = [desc[0] for desc in curs.description] while True: print('-' * 30) row = curs.fetchone() if not row: break for (name, value) in zip(colnames, row): print('%s => %s' % (name, value)) conn.commit()# Save inserted records# Fetch and open/play a file by FTPimport webbrowser, sys from ftplib import FTP# Socket-based FTP toolsfrom getpass import getpass# Hidden password inputif sys.version[0] == '2': input = raw_input# 2.X compatibilitynonpassive = False# Force active mode FTP for server?filename = input('File?')# File to be downloadeddirname = input('Dir? ') or '.'# Remote directory to fetch fromsitename = input('Site?')# FTP site to contactuser = input('User?')# Use () for anonymousif not user: userinfo = () else: from getpass import getpass# Hidden password inputuserinfo = (user, getpass('Pswd?')) print('Connecting...') connection = FTP(sitename)# Connect to FTP siteconnection.login(*userinfo)# Default is anonymous loginconnection.cwd(dirname)# Xfer 1k at a time to localfileif nonpassive:# Force active FTP if server requiresconnection.set_pasv(False) print('Downloading...') localfile = open(filename, 'wb')# Local file to store downloadconnection.retrbinary('RETR ' + filename, localfile.write, 1024) connection.quit() localfile.close() print('Playing...') webbrowser.open(filename)